Senior Data Quality Engineer, ML

WHAT YOULL DO

• Write Python and SQL scripts to evaluate outputs from large language models (LLMs)
• Design and implement LLM-as-Judge evaluations with clear scoring rubrics (faithfulness, relevance, completeness, correctness)
• Define and calculate quality metrics such as exact match, token-level F1, ROUGE, and subjective rubric scores
• Build and maintain ground-truth datasets for benchmarking and regression testing
• Automate evaluation pipelines and integrate them into CI/CD workflows
• Conduct in-depth analysis of large unstructured datasets to identify inconsistencies, anomalies, missing values, and potential biases
• Diagnose and report failure modes (hallucinations, irrelevant answers, formatting errors)
• Collaborate and serve as a crucial link between AI engineers, QA, data scientists and product managers to set quality standards and release criteria
• Document processes and maintain reproducibility of evaluation runs
• Create comprehensive technical documentation, including design specifications, architecture diagrams, and code comments

WHAT YOULL NEED

• 6-8 years of experience in data quality engineering, with hands-on expertise in Python, SQL, automated evaluation pipelines, LLM quality metrics, and end-to-end data validation across complex datasets and AI/ML systems
• Strong proficiency in Python and SQL (data handling, scripting, test automation)
• Experience with data cleaning and standardization techniques to facilitate ingestion and analysis by various teams
• Understanding of generative AI concepts (prompts, hallucinations, grounding)
• Experience designing structured LLM prompts for evaluations
• Familiarity with at least one evaluation framework (RAGAS, DeepEval, TruLens, LangSmith) or ability to learn quickly
• Familiarity with cloud runs and automation (GCP preferred) or ability to learn quickly
• Ability to translate ambiguous quality expectations into measurable metrics
• Excellent problem-solving abilities and analytical thinking
• Effective communication skills to collaborate with cross-functional teams and present technical concepts to both technical and non-technical stakeholders.