Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Home
>
Jobs in andhra pradesh
>
Virtusa
>
Automation & Benchmarking Engineer

Automation & Benchmarking Engineer

Virtusa

8 years

0 Lacs

andhra pradesh india

Posted:2 days ago| Platform:

Apply

Skills Required

automation benchmarking automate dataset evaluation ai software design aggregation tooling developer code visualization reports looker python logging data management deployment compliance gitlab bitbucket engineering test javascript airflow rest integration metrics tableau gcp github ml model

Work Mode

On-site

Job Type

Full Time

Job Description

Role OverviewWe are seeking

Automation & Benchmarking Engineers

to build scalable pipelines and tools that automate dataset ingestion, evaluation, and benchmarking of AI systems.This role blends

software engineering

automation scripting

, and

AI performance analysis

to accelerate evaluation workflows and generate insights across Google’s AI tools.Key Responsibilities

Design and develop end-to-end automation pipelines for evaluation workflows—prompt submission, response collection, result aggregation, and reporting.
Integrate evaluation tooling with developer surfaces like Gemini CLI, VS Code, and GitHub.
Conduct competitive benchmarking against peer AI tools to measure correctness, verbosity, and usefulness.
Build dashboards and visualization reports using Looker Studio, BigQuery, or Python-based tools.
Optimize system performance, automate error logging, and maintain reproducibility across evaluations.
Collaborate with TPM and data specialists to deliver evaluation automation at scale.
Ensure source code management and deployment compliance in GitLab / Bitbucket environments.

Required Skills & Experience

4–8 years of experience in software engineering, test automation, or AI evaluation tooling.
Proficiency in Python, JavaScript, or Go for automation and data handling.
Experience building pipelines or automation frameworks (Airflow, Beam, or custom orchestration).
Strong understanding of REST APIs, LLM integration, and evaluation metrics computation.
Hands-on experience with data visualization tools (Looker, Tableau, Plotly).
Familiarity with cloud platforms (GCP preferred) and source control workflows (GitHub/GitLab).

Preferred Qualifications

Experience benchmarking AI-assisted developer tools (e.g., Copilot, TabNine, Replit).
Knowledge of ML model evaluation metrics and comparative performance analysis.
Background in computer science, applied ML, or automation frameworks.

More Jobs at Virtusa

Servicenow QA

Bengaluru, Karnataka

3 - 3 yrs

Salary: Not disclosed

React T3

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Java-Pankaj

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

C ++

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Java Architect (ATC)

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.