Automation & Benchmarking Engineer

Automation & Benchmarking Engineer

Role Overview

We are seekingAutomation & Benchmarking Engineersto build scalable pipelines and tools that automate dataset ingestion, evaluation, and benchmarking of AI systems.
This role blendssoftware engineering,automation scripting, andAI performance analysisto accelerate evaluation workflows and generate insights across Google’s AI tools.

Key Responsibilities

• Design and developend-to-end automation pipelinesfor evaluation workflows—prompt submission, response collection, result aggregation, and reporting.
• Integrate evaluation tooling with developer surfaces likeGemini CLI, VS Code, and GitHub.
• Conductcompetitive benchmarkingagainst peer AI tools to measure correctness, verbosity, and usefulness.
• Build dashboards and visualization reports usingLooker Studio, BigQuery, or Python-based tools.
• Optimize system performance, automate error logging, and maintain reproducibility across evaluations.
• Collaborate with TPM and data specialists to deliver evaluation automation at scale.
• Ensure source code management and deployment compliance inGitLab / Bitbucketenvironments.

Required Skills & Experience

• 4–8 years of experience insoftware engineering, test automation, or AI evaluation tooling.
• Proficiency inPython, JavaScript, or Gofor automation and data handling.
• Experience building pipelines or automation frameworks (Airflow, Beam, or custom orchestration).
• Strong understanding ofREST APIs,LLM integration, andevaluation metrics computation.
• Hands-on experience withdata visualization tools(Looker, Tableau, Plotly).
• Familiarity withcloud platforms (GCP preferred)andsource control workflows(GitHub/GitLab).

Preferred Qualifications

• Experience benchmarking AI-assisted developer tools (e.g., Copilot, TabNine, Replit).
• Knowledge ofML model evaluation metricsandcomparative performance analysis.
• Background in computer science, applied ML, or automation frameworks.