Manager QA Evaluation / AI Evaluation Engineer

Manager QA Evaluation / AI Evaluation Engineer

The role offers an opportunity to build the AI evaluation function from the ground up, defining how AI agents are tested, measured, and validated for reliability, safety, and production readiness.

Key highlights of the role:

• Architect evaluation frameworks for LLMs and multi-step AI agents.
• Define acceptance criteria, quality benchmarks, and release gates for AI systems.
• Evaluate agent reasoning, tool orchestration, behavioral consistency, and failure patterns.
• Establish production monitoring, model regression testing, drift detection, and AI safety evaluation.
• Influence technical architecture and AI quality standards across engineering teams.

What were looking for:

• 10+ years of software engineering experience, including 4+ years of senior individual-contributor leadership in AI evaluation, measurement infrastructure, or AI quality systems.
• Strong hands-on experience with LLM evaluation, agentic AI, evaluation frameworks, and production AI systems.
• Deep understanding of distributed systems, statistical evaluation, and production reliability.
• Experience defining technical standards and influencing architecture across teams.

Role level:

Location:

Best regards,
Amreesh Yadav