AI Agent Quality Engineering

About the job

About Client:

Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation. It partners with major firms in banking, healthcare, telecom, and media.

Our Client is known for combining deep industry expertise with agile development practices, enabling scalable and cost-effective digital transformation. The company operates in over 50 locations across more than 25 countries, has delivery centers in Asia, Europe, and North America and is backed by Baring Private Equity Asia.

Job Title : AI Agent Quality Engineering

Key Skills : AI Agent,AI, Python,LLM,Automate regression,A/B Testing,hallucination

Experience : 5-7 Years

Location: PAN INDIA

Education Qualification : Any Graduation

Work Mode : Hybrid

Employment Type : Contract to Hire

Notice Period : Immediate - 10 Days.

Test Engineer to validate agent output quality, automate regression, and perform hallucination testing

• Design and execute functional and exploratory test cases.

• Validate LLM outputs and context alignment.

• Automate regression tests using Python frameworks.

• Track success rate, response time, hallucination %.

• Perform agent A/B testing and red teaming.

• Experience with AI/ML or chatbot testing.

• Familiarity with Pytest, Robot Framework.

• Knowledge of LLM output evaluation techniques.

• Experience with annotation and human-in-the-loop testing.