Associate Staff Engineer, Python QA

Company Description

We're Nagarro.

REQUIREMENTS:

• Total Experience: 5+ years
• Strong expertise in

  Python-based QA automation

  for backend, API, and AI-driven systems.
• Hands-on experience building automation frameworks using

  Python and/or JavaScript

  .
• Strong understanding of

  API automation testing

  (REST, JSON, authentication, validation).
• Experience with

  Selenium

  for UI automation (good to have).
• Hands-on experience testing

  AI APIs, LLM-based systems, agentic workflows, and RAG pipelines

  .
• Experience implementing

  LLM evaluation metrics

  such as accuracy, consistency, latency, and performance.
• Strong understanding of

  test design techniques, regression strategies, and reliability testing

  .
• Experience validating AI model behavior, edge cases, hallucination risks, and response consistency.
• Familiarity with

  ML pipelines, prompt testing, embeddings, vector databases

  , or inference workflows (preferred).
• Experience with

  test data creation and management

  for AI and API testing.
• Exposure to

  performance and load testing

  of APIs and AI services. Strong understanding of CI/CD integration for automated test execution.
• Experience working in

  Agile/Scrum

  environments with cross-functional teams. Ability to analyze logs, metrics, and traces for debugging test failures.
• Excellent communication, analytical, and problem-solving skills.

RESPONSIBILITIES:

• Design, develop, and maintain

  automation test suites

  for AI APIs, LLM services, agentic workflows, and RAG systems.
• Build scalable and maintainable

  Python-based automation frameworks

  for backend and AI testing.
• Implement and execute

  LLM evaluation tests

  covering accuracy, consistency, reliability, and performance benchmarks.
• Develop automated tests for

  REST APIs

  , validating request/response schemas, data integrity, and error handling.
• Collaborate closely with

  ML engineers, backend developers, and full-stack teams

  to validate releases.
• Contribute to

  regression, reliability, and performance test coverage

  for AI-driven platforms.
• Validate AI system behavior across different prompts, inputs, and edge cases.
• Ensure automated test coverage for model updates, prompt changes, and pipeline enhancements.
• Integrate automation suites with

  CI/CD pipelines

  for continuous testing.
• Participate in requirement analysis and define

  test strategies for AI-first systems

  .
• Analyze failures, identify root causes, and provide actionable feedback to development teams.
• Support release readiness through comprehensive automated regression cycles.
• Maintain clear documentation of test cases, frameworks, and evaluation metrics.
• Continuously improve automation effectiveness by adopting new tools, techniques, and best practices.
• Mentor junior QA engineers on Python automation and AI testing concepts.