AI Evaluation & Data Engineering Specialist

• Build and maintain evaluation datasets for AI models across programming languages (Python, Golang, JavaScript, Java).
• Develop and apply data labeling and scoring guidelines based on Google’s evaluation framework.
• Implement LLM-judge calibration workflows to align automated and human evaluations.
• Perform error analysis, drift detection, and regression testing of AI model outputs.
• Collaborate with automation engineers to integrate datasets into evaluation pipelines.
• Support rater training, inter-rater reliability checks, and dataset validation reviews.
• Manage data quality assurance and documentation for contributions to Google-maintained repositories.
  

Required Skills & Experience

• 4–8 years of experience in AI/ML data operations, evaluation, or data engineering.
• Proficiency in Python (mandatory) for dataset manipulation, analysis, and scripting.
• Experience with LLM evaluation, prompt engineering, or text generation quality assessment.
• Familiarity with Gemini CLI, Vertex AI, or LangChain evaluation tools.
• Strong understanding of data curation, annotation workflows, and labeling quality metrics.
• Hands-on with Git-based repositories and CI/CD data workflows.
• Excellent analytical and problem-solving skills with attention to detail.
  

Preferred Qualifications

• Experience evaluating code-generation or NLP-based AI products.
• Exposure to data governance and privacy compliance frameworks.
• Background in computer science, data science, or linguistics preferred