Gen AI Engineering in Test

Primary Title: Generative AI Engineer in Test

Industry & Sector:

Operating in the Enterprise Artificial Intelligence and Quality Engineering sector, delivering production-grade generative-AI features, LLM-driven services, and automated validation frameworks for enterprise customers. We build robust evaluation and test automation to ensure safe, reliable, and scalable AI deployments across mission-critical domains.

About The Opportunity

We are hiring an on-site Generative AI Engineer in Test to join a fast-paced engineering organisation that embeds testing and quality practices into the ML/GenAI lifecycle. You will design, implement, and operate test automation and evaluation pipelines that validate model behavior, prompt robustness, safety, and integration with cloud services.Role & Responsibilities

• Design and implement automated test suites for generative models: unit, integration, regression, and adversarial tests focused on hallucination, bias, safety, and output consistency.
• Build and maintain prompt-evaluation frameworks and metric-driven validators for response quality, factuality, and latency across LLM endpoints.
• Integrate model tests into CI/CD pipelines to enable pre-release gating, canary evaluations, and automated rollback criteria.
• Create reproducible test harnesses for model fine-tuning runs, inference endpoints, and vector-search workflows; automate data generation and evaluation artifacts.
• Develop observability and alerting for model drift, performance regressions, and test flakiness; collaborate with SRE/ML engineers to remediate root causes.
• Document testing playbooks, establish test-data management practices, and mentor engineering teams on GenAI QA best practices.
  

Skills & Qualifications

Must-Have

• Proven experience testing ML/LLM systems and implementing test automation.
• Strong Python coding for test harnesses and automation scripts.
• Hands-on with PyTest or equivalent testing frameworks.
• Experience in prompt engineering and prompt robustness evaluation.
• Familiarity with LangChain or similar orchestration libraries.
• Practical experience integrating tests into CI/CD (Jenkins, GitHub Actions, or equivalent).
  

Preferred

• Experience with OpenAI API and Hugging Face transformers / inference tooling.
• Containerisation experience with Docker and orchestration familiarity (Kubernetes desirable).
• Knowledge of vector databases and retrieval-evaluation patterns (FAISS, Pinecone, etc.).
  

Qualifications

• Relevant engineering degree or equivalent hands-on experience in software/ML testing.
• 3+ years in test automation or quality engineering with at least 1+ years working on ML/AI model validation preferred.
  

Location & Workplace
On-site role India. Candidates must be willing to work from the designated office location.Benefits & Culture Highlights

• Collaborative, engineering-first culture with strong emphasis on learning and technical ownership.
• Competitive compensation with performance-linked incentives and learning budget for conferences/certifications.
• Opportunity to shape GenAI QA practices and influence production-grade AI deployments.
  

To apply, please submit your resume and examples of test frameworks, evaluation scripts, or repos demonstrating GenAI testing experience. This role is sourced by Viraaj HR Solutions and is an on-site opportunity across India.
Skills: python,gen ai,testing,gen ai engineering,docker,api testing