AI Quality Assurance Lead

It's exciting to be part of a team where everyone truly believes in the company's mission and values! At our company, we are dedicated to bringing enthusiasm and a customer-centric approach to our work. As an AI Quality Assurance Lead specializing in Evaluation & Testing, you will be based in our Hyderabad office 2-3 days a week. You will play a crucial role in overseeing ABC Fitness's evaluation frameworks, testing pipelines, and compliance tools for AI-driven features. Collaborating with product, engineering, and legal teams, you will ensure that every interaction with our Generative AI technology meets the highest standards of accuracy, safety, and performance. Your responsibilities will include designing hybrid testing systems, defining quality KPIs for GenAI, and integrating Responsible AI principles throughout our fitness-tech platform. At ABC Fitness, we have a strong entrepreneurial spirit, which drives us to roll up our sleeves, act swiftly, and learn together. We value individuals who are willing to take on challenges and grow alongside us. In this role, you will: - Design and implement evaluation pipelines for generative AI systems using tools such as OpenAI Evals, Promptfoo, and custom test harnesses. - Develop workflows for detecting hallucinations and frameworks for analyzing biases in LLM outputs across multiple languages. - Collaborate with AI researchers to translate model capabilities into testable requirements for product teams. - Establish CI/CD-integrated regression testing for AI microservices on cloud platforms like AWS/Azure, monitoring model performance and drift. - Lead bug triage sessions, prioritizing issues that impact user trust, legal compliance, or revenue. - Document QA protocols, failure modes, and root-cause analyses in our internal knowledge base. To be successful in this role, you will need: - 7+ years of experience in QA/testing roles, with a focus on AI/ML systems for at least 3 years (LLMs, recommendation engines, conversational AI). - Hands-on experience with GenAI evaluation tools like LangSmith, Weights & Biases, and proficiency in statistical analysis using Python and SQL. - Proficiency in cloud platforms such as AWS SageMaker, Azure ML, and containerized testing environments like Docker and Kubernetes. - A deep understanding of Responsible AI principles, including fairness, transparency, privacy, and experience with adversarial testing methodologies. - The ability to mentor junior engineers and effectively communicate technical risks to non-technical stakeholders. - Certifications like AWS Certified Machine Learning Specialty or Microsoft AI Engineer would be advantageous. At ABC Fitness, we offer: - A purpose-led company with a culture centered around values such as Best Life, One Team, and Growth Mindset. - Competitive PTO plans, including 15 days of accrued leave, 12 days of Sick leave, and 12 days of Casual leave per year. - Group Mediclaim insurance coverage, Life Insurance, Personal Accident Insurance, and other benefits. - Support for working women with financial aid for childcare facilities. - An inclusive and diverse workplace where all employees are valued and encouraged to reach their full potential. If you are passionate about diversity, equality, and inclusion, we invite you to apply for this role at ABC Fitness and be a part of our dynamic team!,