QA Lead

QA Lead

This role requires strong analytical skills, understanding of AI/ML concepts, and the ability to validate both deterministic and non-deterministic system behaviors.

Role and responsibilities

• Define and execute

  AI testing strategies

  including model accuracy, precision/recall validation, bias, drift, confidence scoring, and edge-case analysis.
• Validate outputs generated by LLMs, NLP pipelines, CV/OCR models, or structured ML models.
• Establish test datasets (golden datasets) for benchmarking AI model performance.
• Work closely with Data Scientists and ML Engineers to understand model behavior, input constraints, and expected outcomes.
• Track

  model drift

  , compare historical vs new model outputs, and support continuous model monitoring.

• Lead automation efforts across

  API, UI, data pipelines

  , and

  AI inference workflows

  .
• Build automated test harnesses to validate model outputs at scale.
• Implement regression automation for inference pipelines, vector embeddings, decision logic, and business rule layers.
• Develop dashboards to track AI model KPIs, accuracy trends, false positives/false negatives, and prediction quality.
• Validate training, testing, and inference datasets for completeness, correctness, and anomalies.
• Perform

  cross-validation

  between model outputs and business rules.
• Validate correct functioning of inference APIs, embedding generation, OCR extraction, and AI-driven business workflows.
• Conduct load and performance testing for: AI inference APIs, Model-serving endpoints, Vector embeddings and similarity search, Document ingestion (OCR pipelines).
• Ensure response time SLAs, throughput thresholds, concurrency handling, and API gateway performance.
• Establish evaluation methods for non-deterministic predictions (LLMs, OCR, NLP pipelines).
• Standardize expected-output rules using confidence thresholds, tolerance bands, and scoring matrices.
• Drive alignment with product/architecture teams on acceptable outcome ranges.
• Lead and mentor a team of QA engineers (API, UI, Data, Performance).
• Allocate tasks, manage sprint QA deliverables, and track quality KPIs.
• Drive defect triage, issue root-cause analysis, and production readiness assessments.
• Collaborate closely with Engineering, Data Science, Product, and DevOps teams.