Python QA Analyst - AI & Agentic

Job Description

About the Role We're seeking a skilled Python QA Analyst to join our team focused on AI and agentic systems. In this role, you'll ensure the quality and reliability of our AI products while occasionally supporting agentic development during periods of lower QA workload. Key Responsibilities Design, implement and maintain comprehensive test suites for AI and agentic systems Create automated testing frameworks using Python to validate AI model outputs and behaviors Develop test cases that cover edge scenarios and potential failure modes in AI systems Perform thorough regression testing on new releases and feature updates Validate that AI systems meet functional requirements and performance benchmarks Document bugs with clear reproduction steps and expected behaviors Collaborate with AI engineers to understand model architectures and potential weak points Support the development team in troubleshooting and resolving identified issues Assist with agentic development projects during periods of lower QA workload Requirements Experience in software QA, with a focus on Python Strong Python programming skills and experience with testing frameworks (pytest, unittest) Understanding of AI concepts, machine learning workflows, and LLM evaluation techniques Experience testing complex systems with multiple integrations Familiarity with CI/CD pipelines and test automation Knowledge of version control systems (Git) Experience with bug tracking and project management tools Nice to Have Experience with AI-specific testing methodologies and evaluation metrics Understanding of LLM evaluation techniques (including hallucination detection, bias testing) Background in agentic system development or testing Experience with containerization (Docker) and orchestration tools Knowledge of API testing (Postman, requests) Familiarity with prompt engineering and prompt testing methodologies What You'll Do Develop comprehensive test plans for new AI features and systems Build and maintain automated test suites that validate model outputs against expected benchmarks Create specialized tests for agentic behaviors and decision-making processes Collaborate with cross-functional teams to define quality standards and acceptance criteria Participate in code reviews from a quality assurance perspective Contribute to agentic development projects when QA workload allows Help establish testing best practices for AI and agentic systems This position offers the opportunity to work at the cutting edge of AI quality assurance while developing skills in agentic system development. Join us in building reliable, high-quality AI products that our customers can trust.