Gen AI QA Engineer – AI Consulting & Strategy

Experience : 5+ Years Experience
Education : Bachelor’s/Master’s degree
Work Location : Chennai, India (Chennai/Remote/Hybrid)

Key Responsibilities :

• Design, develop, and maintain automated testing frameworks for Gen AI applications and agentic systems.
• Implement evaluation pipelines to measure and improve LLM reliability and response quality.
• Develop and operationalize hallucination and bias detection systems.
• Collaborate with LLM Engineers to integrate evaluation checkpoints into model deployment workflows.
• Define and manage evaluation datasets and metrics for automated quality reporting.
• Partner with Product and Governance teams to ensure responsible AI testing and validation practices.
• Continuously monitor and improve AI performance, reliability, and trust metrics.

Required Technical Skills :

Generative AI Quality Engineering

• Design automated testing frameworks for generative AI and multi-agent systems.
• Develop evaluation pipelines to assess LLM responses for accuracy, coherence, and factual grounding.
• Implement hallucination detection mechanisms using reference comparison, confidence scoring, or retrieval checks.
• Create reproducible test datasets and evaluation benchmarks for prompt-response validation.

Automation Frameworks & Tools

• Develop and maintain Python-based QA frameworks integrated with CI/CD systems.
• Experience with testing tools such as pytest, LangSmith, DeepEval, or TruLens.
• Automate testing of API-driven AI systems, RAG pipelines, and multi-agent conversations.
• Implement regression, performance, and stress testing for large-scale AI applications.

Evaluation Metrics & Analysis

• Define and compute LLM evaluation metrics (precision, recall, coherence, diversity, factual accuracy, toxicity).
• Build evaluation dashboards and reporting tools for continuous model monitoring.
• Analyze model drift, prompt sensitivity, and performance degradation over time.

Model Safety & Reliability

• Develop and integrate toxicity, bias, and compliance detection checks into QA pipelines.
• Collaborate with security and governance teams to ensure AI safety and alignment standards.
• Implement guardrail testing and policy compliance frameworks for generative systems.

Infrastructure & Integration

• Integrate QA pipelines into CI/CD workflows for LLM deployment pipelines.
• Experience with cloud-native tools (AWS, Azure, GCP, or OCI) for automated test execution.
• Implement test orchestration using Docker, Kubernetes, or serverless workflows.
• Familiarity with API mocking, synthetic test data generation, and version-controlled evaluation artifacts.

Required Skills & Experience :

• 5+ years of experience in QA Automation, AI Testing, or ML Evaluation Engineering.
• Strong programming background in Python with experience in PyTest, FastAPI, or Playwright.
• Understanding of LLM architectures, prompt-response mechanics, and RAG systems.
• Experience with AI evaluation frameworks such as DeepEval, TruLens, or LangSmith.
• Familiarity with data validation, model observability, and pipeline testing.
• Proven ability to design automated evaluation frameworks for AI and ML models.