QA Test Lead - Automation

QA Lead (AI Products)

Experience:

About the role

Youll lead the QA function for our AI-driven products - chatbots (web/app/WhatsApp), IVR voice bots, NLP services, and integrations deployed with enterprise and government clients. Your mandate: build repeatable, data-driven QA practices that keep quality, safety, and compliance high across rapid releases.

What youll do

Own quality strategy & release gates:

Lead the QA team:

AI-specific quality:

• Create & maintain evaluation datasets for intents, entities, RAG responses, OCR accuracy, and IVR call-flows.
• Track key model/product metrics: intent F1, fallback/containment rate, hallucination rate, toxicity/safety violations, WER/CER for ASR/TTS, latency (p95), and CSAT/deflection.
• Run prompt regression, red-teaming for prompt injection/safety, and content policy checks.

Multilingual & channel QA:

API & microservices testing:

Performance & reliability:

Automation first:

Production quality ops:

Compliance & security basics:

Stakeholder management:

Must-haves

• 3-6 years in QA with

  hands-on automation

  + at least 1 year leading/mentoring testers.
• Experience testing

  AI/ML or LLM-based

  products (chatbots, RAG/NLP pipelines, OCR, or voice bots).
• Solid with

  API testing

  and tools like Postman/Newman, PyTest, JSON schema/contract testing.
• UI automation using

  Playwright or Cypress

  ; CI/CD integration experience.
• Familiarity with

  performance testing

  (JMeter/Locust/k6) and reading dashboards/logs (Grafana/ELK/Sentry/DataDog).
• Strong test design (equivalence, boundary, combinatorial, exploratory), defect triage, and release management.
• Excellent communication; comfortable leading client-facing UAT.

Good-to-haves

• Exposure to

  IVR/telephony

  QA (SIP/PSTN basics, DTMF/ASR/TTS evaluation, call-flow testing).

• Multilingual QA

  experience for Indian languages; knowledge of WER/CER measurement.
• Understanding of

  LLM eval frameworks

  (promptfoo, DeepEval/OpenAI Evals/RAGAS) and safety/red-teaming.
• Cloud & containers (AWS/Azure/GCP, Docker/K8s) and environment management.
• Accessibility testing, security testing fundamentals,

  data privacy

  (e.g., DPDP/GDPR-aligned practices).
• Jira, Zephyr/TestRail, Confluence; Browser Stack/device labs.