Mid QA - Automated Testing & AI Validation

Job Title:

Experience:

Employment Type:

Overview

We are seeking a Mid-Level QA Engineer – Automated Testing & AI Validation with hands-on experience in testing AI-powered systems, particularly LLM-driven applications. The ideal candidate will have a strong QA automation background, a growing understanding of probabilistic AI outputs, and enthusiasm for building and improving AI evaluation frameworks in real-world, production environments.

This role requires high autonomy, comfort with ambiguity, and strong written communication skills in an async-first setup.

Key Responsibilities

• Design, develop, and execute automated test cases using Python and Pytest for AI-driven applications.
• Validate LLM integrations, APIs, and multi-agent workflows through functional, regression, and smoke testing.
• Perform intent classification, semantic similarity, and response consistency testing for conversational AI systems.
• Conduct hallucination detection and factual accuracy checks using automated and semi-automated methods.
• Implement response quality scoring using LLM-as-a-Judge evaluation patterns.
• Use LLM observability and tracing tools such as LangFuse or LangSmith to monitor and validate model behavior.
• Test conversational AI applications including chatbots and virtual assistants across use cases.
• Support Kubernetes-based application health checks and basic smoke testing.
• Integrate automated tests into CI/CD pipelines using GitHub Actions.
• Document test cases, evaluation criteria, defects, and QA findings clearly and concisely.
• Collaborate with engineering and AI teams to improve evaluation pipelines and testing strategies.
• Continuously learn and contribute to advanced AI evaluation methodologies.

Skills

• 3–5 years of experience in QA or test automation roles.
• Strong proficiency in Python-based test frameworks (Pytest).
• Basic to intermediate understanding of LLM evaluation concepts and AI system testing.
• Hands-on experience with LLM observability and tracing tools (LangFuse, LangSmith, or similar).
• Experience in API testing and validation of AI integrations.
• Exposure to conversational AI or chatbot testing.
• Familiarity with CI/CD processes and GitHub Actions.
• Knowledge of Kubernetes-based environments for testing and validation.
• Strong documentation and defect reporting skills.
• Ability to understand semantic correctness over strict assertion-based testing.
• Self-driven mindset with the ability to work independently and handle ambiguity.

Preferred Skills

• Experience testing production-grade LLM systems with real user traffic.
• Exposure to LLM-as-a-Judge frameworks and custom evaluation pipelines.
• Familiarity with Azure-based infrastructure (AKS, Key Vault, PostgreSQL).
• Understanding of multi-agent frameworks such as LangGraph or Microsoft Agent Framework.
• Interest in AI ethics, responsible AI, and high-quality evaluation practices.