CoRover Private Limited/BharatGPT

QA Lead (AI Products) Experience: 36 years in QA (at least 1 year leading/mentoring a small team) Domain: Conversational AI/LLM chatbots, IVR/telephony, multilingual NLP, API-first microservices About the role Youll lead the QA function for our AI-driven products - chatbots (web/app/WhatsApp), IVR voice bots, NLP services, and integrations deployed with enterprise and government clients. Your mandate: build repeatable, data-driven QA practices that keep quality, safety, and compliance high across rapid releases. What youll do Own quality strategy & release gates: Define risk-based test plans, traceability to SOW/RFP requirements, and “go/no-go” criteria for every release. Lead the QA team: Coach testers, run daily QA standups, manage workload in Jira/Zephyr/TestRail, and drive defect triage with Product/Engg. AI-specific quality: Create & maintain evaluation datasets for intents, entities, RAG responses, OCR accuracy, and IVR call-flows. Track key model/product metrics: intent F1, fallback/containment rate, hallucination rate, toxicity/safety violations, WER/CER for ASR/TTS, latency (p95), and CSAT/deflection. Run prompt regression, red-teaming for prompt injection/safety, and content policy checks. Multilingual & channel QA: Systematically test Hindi + Indic languages (e.g., Tamil/Telugu), web/app widgets, WhatsApp, IVR (DTMF, ASR/TTS), and SDK/WebView embeds across devices/browsers. API & microservices testing: Automate API tests (functional, contract, schema), resilience tests, and compatibility across environments; validate third-party integrations. Performance & reliability: Execute load/stress tests for peak traffic; monitor SLOs/SLA adherence post-release with observability dashboards. Automation first: Establish and scale automation for UI (Playwright/Cypress) and API (Postman/Newman/PyTest), integrate into CI/CD (GitHub Actions/GitLab/Jenkins), and maintain flake-free pipelines. Production quality ops: Set up canary/synthetic checks, log-based validations, and rapid rollback playbooks; drive RCA and preventive actions. Compliance & security basics: Enforce PII redaction, access controls, and audit trails; ensure WCAG 2.1 AA accessibility and alignment with client data/security requirements. Stakeholder management: Partner with Product, ML, and Engineering; conduct UAT with clients (incl. PSU/enterprise), and present quality reports with clear metrics and risks. Must-haves 3-6 years in QA with hands-on automation + at least 1 year leading/mentoring testers. Experience testing AI/ML or LLM-based products (chatbots, RAG/NLP pipelines, OCR, or voice bots). Solid with API testing and tools like Postman/Newman, PyTest, JSON schema/contract testing. UI automation using Playwright or Cypress ; CI/CD integration experience. Familiarity with performance testing (JMeter/Locust/k6) and reading dashboards/logs (Grafana/ELK/Sentry/DataDog). Strong test design (equivalence, boundary, combinatorial, exploratory), defect triage, and release management. Excellent communication; comfortable leading client-facing UAT. Good-to-haves Exposure to IVR/telephony QA (SIP/PSTN basics, DTMF/ASR/TTS evaluation, call-flow testing). Multilingual QA experience for Indian languages; knowledge of WER/CER measurement. Understanding of LLM eval frameworks (promptfoo, DeepEval/OpenAI Evals/RAGAS) and safety/red-teaming. Cloud & containers (AWS/Azure/GCP, Docker/K8s) and environment management. Accessibility testing, security testing fundamentals, data privacy (e.g., DPDP/GDPR-aligned practices). Jira, Zephyr/TestRail, Confluence; Browser Stack/device labs.