Quality Assurance Engineer (AI)

Join Delphi - Where Innovation meets transformation

hybrid work model

We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.

Data, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization.

About the Role: Sr Consultant - Quality Assurance

Location:

Sr Consultant - Quality Assurance (AI)

What You’ll Do:

• Design and execute

  evaluation strategies for LLMs

  , sub-models, and multi-agent systems.
• Perform

  manual and automated testing

  of AI models, ensuring comprehensive coverage.
• Develop

  synthetic test data using Python scripting

  to simulate real-world scenarios and edge cases.
• Ensure

  boundary condition coverage

  through automated data generation techniques.
• Prepare and maintain

  prompt libraries

  for model evaluation across diverse use cases.
• Evaluate

  multi-model and multi-agent architectures

  , ensuring functional and performance benchmarks are met.
• Apply and interpret

  evaluation metrics

  (e.g., accuracy, precision, recall, F1-score, Bias check, hallucination, Consistency, Relevance, Fact Check, BLEU, ROUGE, perplexity, etc.) with a clear understanding of their formulas and implications.
• Collaborate closely with Data Science and Engineering teams to align QA strategies with model development.
• Document test plans, test cases, and evaluation reports for stakeholder review and signoff.

What You’ll Bring:

• 5+ years of relevant experience in quality assurance and AI/ML evaluation.
• Strong hands-on experience with

  LLM evaluation techniques

  .
• Proficiency in

  Python

  for scripting, test automation, and synthetic data generation.
• Deep understanding of

  AI model architecture

  , including models, sub-models, and agent-based systems.
• Familiarity with

  prompt engineering

  and its role in model testing.
• Experience with

  AI/ML testing frameworks

  and tools.
• Solid grasp of

  evaluation metrics

  and their practical application.
• Excellent analytical, documentation, and communication skills.
• Prior experience in QA for AI/ML products.
• Exposure to tools like RAGAS, DeepEval, Arize Phoenix, OpenAI APIs, or similar.
• Experience with CI/CD pipelines for AI model deployment and testing.
• A sense of humor

What We Offer:

competitive compensation package, performance-based incentives,