QA Automation Engineer

This role is for one of the Weekday's clients

AI Agentic workflows, LLM evaluation, Retrieval-Augmented Generation (RAG) architectures,

PyTorch

Requirements

Key Responsibilities

• Design, develop, and maintain automation frameworks using Python, Selenium (preferred), Rest Assured, and Pytest/JUnit.
• Build and scale automated test suites for web applications, APIs, microservices, and AI/ML pipelines.
• Create effective test strategies, test plans, and robust regression suites aligned with product and release goals.
• Work closely with developers to define clear testability requirements and ensure high-quality code delivery.
• Develop automation for model validation, LLM behavior testing, RAG pipeline consistency, and agentic workflow scenarios.
• Implement data-driven testing approaches for AI/ML systems, incorporating tools such as PyTorch for environment setup and model handling.
• Perform end-to-end functional, integration, performance, and reliability testing for both traditional software components and new AI-driven features.
• Collaborate with Data Science and ML Engineering teams to design validation processes for model response correctness, drift checks, hallucination detection, and guardrail verification.
• Conduct root cause analysis on quality issues and drive corrective actions across engineering teams.
• Continuously enhance automation frameworks and CI/CD integrations to optimize execution times and stability.
• Ensure documentation, reporting, and coverage metrics are thorough, accurate, and continuously improved.
  

Required Skills & Experience

• 8-12 years of proven experience in software QA with a strong focus on automation.
• Expert-level hands-on experience in Python for building automation frameworks and tools.
• Strong experience with Selenium for web automation (preferred) and Rest Assured for API automation.
• Hands-on proficiency with test frameworks such as Pytest or JUnit.
• Working knowledge of PyTorch for interacting with models or ML pipelines.
• Demonstrated experience in testing AI Agentic systems, LLM-based features, and RAG architectures.
• Strong understanding of software testing principles, CI/CD pipelines, Git-based workflows, and Agile methodologies.
• Excellent problem-solving, debugging, and analytical skills with the ability to break down complex systems.
• Strong communication skills to collaborate effectively across teams and influence quality processes.