Gen AI Consultant

Role Purpose

The purpose of this role is to develop multi-module test cases and set up test environments while preparing test evidence and validating system functionality.

Experience : 6-10 Years

Role Summary: We are seeking a GenAI Testing professional with experience in validating Large Language Models (LLMs), Agentic AI systems, RAG applications, and AI-powered business solutions. The candidate will drive quality assurance for GenAI implementations, ensuring accuracy, reliability, security, and business alignment of AI-generated outcomes.

Key Responsibilities

Design and execute test strategies for GenAI, RAG, Copilot, and Agentic AI applications.

Validate AI responses for accuracy, relevance, consistency, bias, toxicity, and hallucinations.

Develop AI-specific test scenarios, prompt libraries, and evaluation frameworks.

Create automated validation approaches for LLM outputs.

Perform prompt engineering and prompt optimization activities.

Test Retrieval-Augmented Generation (RAG) workflows and knowledge-grounding mechanisms.

Validate AI guardrails, compliance requirements, and security controls.

Collaborate with Product Owners, Business SMEs, Data Scientists, and AI Architects.

Define KPIs and quality metrics for AI solution evaluation.

Support GenAI adoption initiatives and recommend QA best practices.

Mandatory Skills

GenAI Testing Concepts

LLM Validation & Evaluation

Prompt Engineering

RAG Testing

Functional & Automation Testing

API Testing

Python

AI Model Evaluation Techniques

Jira / Azure DevOps

Agile Delivery Model

Good to Have

Azure OpenAI / OpenAI

LangChain / LangGraph

MCP (Model Context Protocol)

Agentic AI Platforms

GitHub Copilot

AI Security Testing

Python-based Automation Frameworks