Software Engineer AI/ML QA

bebo Technologies

3 - 5 years

0 Lacs

chandigarh

Posted:5 days ago| Platform: GlassDoor logo

Apply

Skills Required

software ai ml testing test strategies automation consistency usability retrieval design dataset regression evaluation scoring rubrics benchmarking analyze model learning coding python java api selenium rest writing reasoning communication compensation development metrics engineering integration

Work Mode

On-site

Job Type

Part Time

Job Description

AI/ML QA Engineer

We are seeking a highly skilled and experienced AI/ML QA Engineer to join our innovative team. This role involves rigorously testing and validating the outputs of Large Language Models (LLMs), RAG systems, and AI agents. You will be responsible for designing and executing comprehensive AI-specific test strategies, evaluating performance, and ensuring the highest quality of our AI products. A strong understanding of AI/ML concepts and robust automation skills are crucial for success in this challenging and rewarding position.

Job Responsibilities

Test and validate LLM outputs, ensuring accuracy, correctness, completeness, consistency, usability, and hallucination analysis.
Evaluate RAG systems, including retrieval accuracy, document relevance, context construction, and full response generation flows.
Test AI agents and autonomous workflows, validating decision-making, task execution, and error handling.
Design and execute AI-specific test strategies: dataset creation, edge-case testing, adversarial testing, pattern-based testing, and regression validation.
Develop evaluation frameworks, scoring rubrics, and benchmarking models for AI quality assessment.
Analyze large volumes of AI-generated responses to identify patterns, root causes, and issue clusters, instead of isolated defects.
Knowledge of testing conversational AI, workflows, or agent-based systems.
Exposure to vector search tools and embedding quality analysis.
Validate fixes using new examples from the same pattern category, ensuring true model improvement.
Collaborate closely with AI/ML engineers, QA teams, and product managers to improve AI accuracy and performance.
Contribute to continuous improvement of AI QA practices, automation, tools, and evaluation datasets.

Job Requirement

B.Tech or equivalent degree in Computer Science (or related field).
3-5 years of QA experience, with at least 2 years focused on AI/ML testing (LLMs, RAG, AI agents).
Proficiency in Artificial Intelligence, Machine Learning, AI Agents, and Large Language Models (LLMs).
Strong knowledge of QA methodologies, test design, functional/non-functional testing, and defect lifecycle management.
Solid understanding of LLM evaluation, hallucination types, prompt behavior, response scoring, and quality metrics.
Good understanding of RAG concepts including vector databases, embeddings, and retrieval relevance.
Coding proficiency in Python or Java (both preferred).
Hands-on experience in Web/API automation using frameworks like Playwright, Selenium, or REST Assured.
Experience in writing automation scripts, maintaining test frameworks, and integrating test suites into CI/CD pipelines.
Ability to analyze large sets of AI outputs for patterns and systemic issues.
Excellent analytical reasoning, problem-solving, and communication skills.

bebo Technologies offers a competitive compensation package, flexible work arrangements, and opportunities for professional growth and development. This role demands a candidate with 3–5 years of robust QA experience, specifically including at least two years in advanced AI/ML testing across LLMs, RAG systems, and AI agents. A deep understanding of LLM evaluation metrics, hallucination types, and prompt engineering is essential. Candidates should possess strong foundational knowledge in QA methodologies, coupled with practical skills in analyzing complex AI outputs. Proficiency in Python or Java for automation and experience with modern Web/API automation frameworks like Playwright are critical for success in contributing to CI/CD integration and ensuring high-quality AI product releases.

More Jobs at bebo Technologies

React Native Developer

Chandigarh, India

3 - 3 yrs

Salary: Not disclosed

Machine Learning Engineer

Chandigarh, India

Experience: Not specified

Salary: Not disclosed

Generative AI Engineer

Chandigarh, India

5 - 5 yrs

Salary: Not disclosed

Senior React Native + Android Developer

Chandigarh, India

7 - 7 yrs

Salary: Not disclosed

Senior Flutter Developer

Chandigarh, India

7.0 - 7.0 yrs

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.