Jobs
Interviews
Atomberg Technologies logo

Specialist - AI Quality Engineer – LLM Evaluation & Mobile Automation

Atomberg Technologies

4 - 6 years

Pune

Posted:5 days ago

Apply

Job Description

AI Quality Engineer – LLM Evaluation & Mobile Automation

ROLE PURPOSE

We are looking for an AI Quality Engineer to build and scale quality assurance for AI-powered features, including LLM-based assistants, search, recommendations, and document understanding. The role combines automated LLM evaluation, mobile and web test automation, API validation, and performance testing. The engineer will own automation frameworks, integrate quality checks into CI/CD pipelines, and proactively identify model regressions, data issues, and reliability risks.

KEY RESPONSIBILITIES

Build and maintain automated LLM evaluation suites (DeepEval, Ragas, promptfoo, LangSmith).
Evaluate AI responses for accuracy, groundedness, relevance, and refusal behaviour.
Maintain golden datasets and track hallucination rates to catch regressions pre-release.
Validate multi-turn conversational workflows, tool calling, and failure recovery.
Test AI safety and guardrails: prompt injection, jailbreaks, PII leakage, blocked topics.
Monitor AI quality, latency, cost, and drift; convert production failures into regression tests.
Validate AI pipeline data using SQL/AWS Athena and verify data protection controls.
Own the Appium automation framework for Android/iOS, including BLE and Wi-Fi provisioning.
Build web/API automation (Playwright, Selenium, Postman) and run load/performance tests.

MUST-HAVE EXPERIENCE

4-6 years in test automation with hands-on ownership of frameworks and test strategy.
Practical experience with an LLM evaluation framework (DeepEval, Ragas, promptfoo, LangSmith).
Strong Python skills with solid OOP concepts and reusable test frameworks.
Hands-on Appium experience for Android and/or iOS automation.
Understanding of non-deterministic AI outputs and evaluation beyond simple pass/fail.
Experience with Playwright or Selenium for web automation.
Proficiency in REST API testing, Swagger/OpenAPI, Postman, and SQL for data validation.
Working knowledge of Git, CI/CD pipelines, and continuous testing workflows.
Working knowledge of AWS (S3, Lambda, CloudWatch, IAM) and experience with AWS Device Farm.

Skills Required

Software testing Artificial Intelligence Injection Quality Safety Selenium Appium Mobile Automation

Learn the skills this job asks for

See your skill gap for Machine Learning Engineer →

Free courses, interview questions and practice for each skill.

Work Mode

Work from Office

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Artificial Intelligence Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Atomberg Technologies logo
Atomberg Technologies

RecommendedJobs for You

Atomberg Technologies logo

Specialist - AI Quality Engineer – LLM Evaluation & Mobile Automation

Atomberg Technologies

Pune

Atomberg Technologies logo

Specialist - AI Quality Engineer – LLM Evaluation & Mobile Automation

Atomberg Technologies

Pune