Lead QA – Automated Testing & AI Validation

Lead QA – Automated Testing & AI Validation

Experience: 6+ Years

Employment Type: Contract

Notice Period: Immediate

Role Overview

We are seeking a Lead QA – Automated Testing & AI Validation with strong expertise in Python automation, LLM evaluation, RAG pipelines, observability, adversarial testing, and Azure monitoring. This role will lead quality assurance initiatives for cutting-edge Generative AI and Multi-Agent Systems.

Key Responsibilities

Lead and scale test automation initiatives with a focus on GenAI and AI systems

Design and execute LLM evaluation frameworks using:

LLM-as-a-Judge (G-Eval, custom evaluators)

Hallucination detection, faithfulness, relevance, precision/recall

Implement RAG evaluation frameworks (RAGAS or similar)

Build Python-based automation frameworks using PyTest & DeepEval

Integrate automation into CI/CD pipelines using GitHub Actions

Design and validate multi-agent evaluation pipelines (tool usage, collaboration, reasoning chains)

Perform adversarial and red-team testing:

Prompt injection

Jailbreak attacks

Bias and toxicity detection

Conduct API testing for microservices (REST, async workflows)

Monitor applications using Azure Application Insights & Log Analytics

Define automated scoring systems for GenAI outputs

Manage synthetic datasets and golden datasets for AI validation

Implement observability and trace monitoring using LangFuse, LangSmith, or similar tools

Mandatory Skills

Test Automation – Python, PyTest, DeepEval

LLM Evaluation – G-Eval, Custom Evaluators, LLM-as-a-Judge

RAG Evaluation – RAGAS, Retrieval Metrics

Evaluation Metrics – Hallucination, Faithfulness, Relevance, Precision/Recall

Observability & Monitoring – LangFuse, LangSmith

CI/CD – GitHub Actions

Multi-Agent Testing – Reasoning & Tool Validation

Adversarial/Red Team Testing – Prompt Injection, Jailbreak, Bias/Toxicity

API Testing – REST & Async Workflows

Azure Monitoring – App Insights, Log Analytics

Synthetic & Golden Dataset Management

Automated Scoring System Design for GenAI Outputs