Senior Quality Assurance Engineer

Company overview:

Opportunity with one of our clients who uses a GenAI-powered companion platform

Role overview:

Must-Have Skills & Experience:

● 4+ years in QA or Software Engineering, with at least 2 years working on AI or LLM-powered

products

● Experience building or evaluating Agentic and LLM-driven tools (e.g., Copilot-like features,

document generation, test automation, prompt evaluation)

● Core QA and Testing Experience

○ Strong understanding of QA principles, software test design, and test automation

○ Experience with test frameworks like PyTest, JUnit, Selenium, etc.

○ Familiarity with static/dynamic analysis tools

● AI/LLM Evaluation

○ Experience with AI-generated content evaluation techniques

○ Understanding of LLMs (OpenAI, Claude, etc.), prompt engineering, and outputs

○ Familiarity with metrics for text/code generation: BLEU, ROUGE, CodeBLEU,

METEOR, etc.

○ Knowledge of hallucination detection and alignment issues

● Programming & Automation

○ Proficiency in Python (especially for scripting AI evaluations)

○ Familiarity with code transformation, AST parsing, linters, etc.

○ Experience with REST APIs, CLI tools, and writing modular scripts

● MLOps / CICD

○ Exposure to MLOps stacks (MLflow, Weights & Biases, etc.)

○ Working knowledge of CI/CD tools (GitHub Actions, Jenkins, etc.)

● Excellent problem-solving, analytical, and stakeholder management skills.

● Ability to thrive in a fast-paced, engineering-driven environment.

● Excellent communication and collaboration skills.

Great to Have:

● Background in software engineering, data science, or cloud technologies.

● Experience working with early-stage startups or fast-paced product engineering teams.

● Experience using tools like Jira, Confluence for backlog and test management.

What would you do here?

As an AI QA Engineer, you will design and Architect a QA Evaluation Framework for evaluating AI-generated artifacts, including Code transformation (modernization), Auto-generated documentation, and AI-synthesized test cases. This will include defining Evaluation layers, defining Metrics, Automation Tooling, and Integration with MLOps and CI/CD, Cross-functional collaboration with AI/ML Engineers, Platform Engineers, and Product SMEs.

What’s in it for you?

● Opportunity to work on cutting-edge technologies like Generative AI, across Data Engineering Cloud, and App Development.

● Work on high-impact product development for both startups and enterprise clients.

● Gain experience in end-to-end product development, collaborating with top-tier engineering teams.

● Exposure to a fast-paced, innovation-driven environment with complex technical challenges.

● Work with seasoned professionals and domain experts in a highly collaborative culture.

● Competitive compensation, learning opportunities, and career growth in a high-end product engineering firm