Prompt Engineer - Data Science & Quality Analysis - India

Requirements

• Prompt Design & Evaluation - Develop, test, and refine prompts for tasks such as text generation, question answering, data classification, and structured data extraction to optimize Voice AI performance
• Data-Driven Analysis & Quality Measurement - Design evaluation frameworks and analyze prompt outputs using quantitative metrics, human-in-the-loop evaluation, and user feedback to identify improvement opportunities
• Experimentation & Iteration - Conduct experiments to test prompt variations, measure their business and operational impact, and iterate to enhance accuracy, consistency, and safety
• Regression Testing & Compliance - Build principled regression test suites using tools like LangFuse and Galileo to ensure prompts remain compliant and high-performing as models and use cases evolve
• Collaboration Across Teams - Work closely with data science, product, legal, engineering, and operations teams to align prompt designs with business goals, operational workflows, and compliance requirements
• Model Adaptation & Strategy Develop - prompts across multiple LLMs (GPT, LLaMA, Gemini, and Checkmate's fine-tuned models), understanding model differences to optimize outputs effectively
• Team Leadership & Mentorship Lead - a team of analysts focused on prompt evaluation and data quality analysis, guiding prioritization, experimentation, and reporting. Collaborate with ops teams for seamless deployment and feedback loops
• Research & Continuous Learning -Stay up to date on emerging prompting techniques, LLM behaviors, evaluation frameworks, and AI safety practices to keep Checkmate's AI solutions best-in-class.
  

Minimum Qualifications

• Strong analytical and data science skills, with hands-on experience in Python (pandas, NumPy, scikit-learn)
• Experience designing and conducting experiments and evaluations in applied AI or NLP contexts
• Proficiency in SQL and working with relational databases (e.g. MySQL, PostgreSQL, Oracle, MS SQL)
• Good understanding of data processing, quality measurement, and testing fundamentals
• Experience leading analyst or operations teams, with strong prioritization, mentorship, and collaboration skills
• Strong problem-solving mindset with a drive to explore, optimize, and automate workflows
• Excellent communication skills for presenting insights to technical and non-technical stakeholders
• Bachelor's degree in Data Science, Computer Science, Statistics, Engineering, or a related field
• Flexible to work US hours until at least 6 p.m. ET, with a strong remote setup
  

Preferred Qualifications

• Experience with LLM evaluation and prompt engineering workflows
• Familiarity with tools like LangFuse and Galileo for prompt evaluation and analysis
• Knowledge of cloud platforms (AWS, GCP, Azure) and data pipeline tools
• Familiarity with machine learning concepts and NLP workflows
• Master's or PhD in Data Science, Statistics, Computer Science, or a related field