AI Quality Assurance Lead

This role requires working from our local Hyderabad office 2-3x a week.
Location: Hyderabad, Telangana, India

ABOUT THE TEAM

The Generative AI Quality & Safety team owns ABC Fitnesss evaluation frameworks, testing pipelines, and compliance tooling for AI-driven features. We partner with product, engineering, and legal teams to ensure every LLM interaction meets rigorous standards for accuracy, safety, and performance. As our AI Quality Assurance Lead, youll architect hybrid (automated + human) testing systems, define GenAI quality KPIs, and embed Responsible AI principles across our fitness-tech platform.

At ABC Fitness, we love entrepreneurs because we are entrepreneurs. We know how much grit it takes to start your own business and grow it into something that lasts. We roll our sleeves up, we act fast, and we learn together.

WHAT YOULL DO

• Design and deploy evaluation pipelines for generative AI systems using tools like OpenAI Evals, Promptfoo, and custom test harnesses.
• Develop hallucination detection workflows and bias-analysis frameworks for LLM outputs across 10+ languages.
• Partner with AI researchers to translate model capabilities into testable requirements for product teams.
• Implement CI/CD-integrated regression testing for AI microservices on AWS/Azure, monitoring model drift and performance degradation.
• Lead bug triage sessions, prioritizing issues impacting user trust, legal compliance, or revenue.
• Document QA protocols, failure modes, and root-cause analyses in our internal knowledge base.

WHAT YOULL NEED

• 7+ years in QA/testing roles, with 3+ years focused on AI/ML systems (LLMs, recommendation engines, or conversational AI).
• Hands-on experience with GenAI evaluation tools (LangSmith, Weights & Biases) and statistical analysis (Python, SQL).
• Proficiency in cloud platforms (AWS SageMaker, Azure ML) and containerized testing environments (Docker, Kubernetes).
• Deep understanding of Responsible AI principlesfairness, transparency, privacyand adversarial testing methodologies.
• Ability to mentor junior engineers and communicate technical risks to non-technical stakeholders.
• Certifications like AWS Certified Machine Learning Specialty or Microsoft AI Engineer are a plus.

WHATS IN IT FOR YOU:

• Purpose led company with a Values focused culture Best Life, One Team, Growth Mindset

• Time Off competitive PTO plans with 15 Earned accrued leave, 12 days Sick leave, and 12 days Casual leave per year

• 11 Holidays plus 4 Days of Disconnect once a quarter, we take a collective breather and enjoy a day off together around the globe. #oneteam

• Group Mediclaim insurance coverage of INR 500,000 for employee + spouse, 2 kids, and parents or parent-in-laws, and including EAP counseling

• Life Insurance and Personal Accident Insurance

• Best Life Perk we are committed to meeting you wherever you are in your fitness journey with a quarterly reimbursement

• Premium Calm App enjoy tranquility with a Calm App subscription for you and up to 4 dependents over the age of 16

• Support for working women with financial aid towards crche facility, ensuring a safe and nurturing environment for their little ones while they focus on their careers.