Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Home
>
Jobs in hyderabad
>
Virtusa
>
Automation & Benchmarking Engineer

Automation & Benchmarking Engineer

Virtusa

4 - 8 years

5 - 9 Lacs

hyderabad

Posted:15 hours ago| Platform:

Apply

Skills Required

automation rest python github test automation error logging airflow ai cloud platforms llm bitbucket javascript tableau model evaluation apis gcp compliance looker gitlab software engineering bigquery data visualization

Work Mode

Work from Office

Job Type

Full Time

Job Description

Automation & Benchmarking Engineer (2 Roles)

Role Overview

We are seeking

Automation & Benchmarking Engineers

to build scalable pipelines and tools that automate dataset ingestion, evaluation, and benchmarking of AI systems. This role blends

software engineering

automation scripting

, and

AI performance analysis

to accelerate evaluation workflows and generate insights across Googles AI tools.

Key Responsibilities

Design and develop
end-to-end automation pipelines
for evaluation workflowsprompt submission, response collection, result aggregation, and reporting.
Integrate evaluation tooling with developer surfaces like
Gemini CLI, VS Code, and GitHub
.
Conduct
competitive benchmarking
against peer AI tools to measure correctness, verbosity, and usefulness.
Build dashboards and visualization reports using
Looker Studio, BigQuery, or Python-based tools
.
Optimize system performance, automate error logging, and maintain reproducibility across evaluations.
Collaborate with TPM and data specialists to deliver evaluation automation at scale.
Ensure source code management and deployment compliance in
GitLab / Bitbucket
environments.

Required Skills & Experience

4-8 years of experience in
software engineering, test automation, or AI evaluation tooling
.
Proficiency in
Python, JavaScript, or Go
for automation and data handling.
Experience building pipelines or automation frameworks (Airflow, Beam, or custom orchestration).
Strong understanding of
REST APIs
,
LLM integration
, and
evaluation metrics computation
.
Hands-on experience with
data visualization tools
(Looker, Tableau, Plotly).
Familiarity with
cloud platforms (GCP preferred)
and
source control workflows
(GitHub/GitLab).

Preferred Qualifications

Experience benchmarking AI-assisted developer tools (e.g., Copilot, TabNine, Replit).
Knowledge of
ML model evaluation metrics
and
comparative performance analysis
.
Background in computer science, applied ML, or automation frameworks.

More Jobs at Virtusa

Servicenow QA

Bengaluru, Karnataka

3 - 3 yrs

Salary: Not disclosed

React T3

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Java-Pankaj

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

C ++

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Java Architect (ATC)

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.