Test Automation Lead (AI ML exp)

Role Overview

As a Test Automation Lead at Dailoqa, you’ll architect and implement robust testing frameworks for both software and AI/ML systems. You’ll bridge the gap between traditional QA and AI-specific validation, ensuring seamless integration of automated testing into CI/CD pipelines while addressing unique challenges like model accuracy, GenAI output validation, and ethical AI compliance. 

Key Responsibilities

Test Automation Strategy & Framework Design

• Design and implement scalable test automation frameworks for 

  frontend (UI/UX)

  , 

  backend APIs

  , and 

  AI/ML model-serving endpoints

   using tools like Selenium, Playwright, Postman, or custom Python/Java solutions. 
• Build 

  GenAI-specific test suites

   for validating prompt outputs, LLM-based chat interfaces, RAG systems, and vector search accuracy. 
• Develop 

  performance testing strategies

   for AI pipelines (e.g., model inference latency, resource utilization). 

Continuous Testing & CI/CD Integration

• Establish and maintain 

  continuous testing pipelines

   integrated with GitHub Actions, Jenkins, or GitLab CI/CD. 
• Implement 

  shift-left testing

   by embedding automated checks into development workflows (e.g., unit tests, contract testing). 

AI/ML Model Validation

• Collaborate with data scientists to test AI/ML models for 

  accuracy

  , 

  fairness

  , 

  stability

  , and 

  bias mitigation

   using tools like TensorFlow Model Analysis or MLflow. 
• Validate 

  model drift

   and retraining pipelines to ensure consistent performance in production. 

Quality Metrics & Reporting

• Define and track KPIs: 

• Test coverage

   (code, data, scenarios) 

• Defect leakage rate

   

• Automation ROI

   (time saved vs. maintenance effort) 

• Model accuracy thresholds

   
• Report risks and quality trends to stakeholders in sprint reviews. 
• Drive adoption of AI-specific testing tools (e.g., LangChain for LLM testing, Great Expectations for data validation). 

Technical Requirements

Must-Have

• 7–10 years in test automation, with 

  2+ years validating AI/ML systems

  . 
• Expertise in: 

• Automation tools

  : Selenium, Playwright, Cypress, REST Assured, Locust/JMeter 

• CI/CD

  : Jenkins, GitHub Actions, GitLab 

• AI/ML testing

  : Model validation, drift detection, GenAI output evaluation 

• Languages

  : Python, Java, or JavaScript 
• Certifications: ISTQB Advanced, CAST, or equivalent. 
• Experience with 

  MLOps tools

  : MLflow, Kubeflow, TFX 
• Familiarity with 

  vector databases

   (Pinecone, Milvus) and 

  RAG workflows

  . 
• Strong programming/scripting experience in JavaScript, Python, Java, or similar 
• Experience with API testing, UI testing, and automated pipelines 

• Understanding of AI/ML model testing, output evaluation, and non-deterministic behavior validation 

• Experience with testing AI chatbots, LLM responses, prompt engineering outcomes, or AI fairness/bias 

• Familiarity with MLOps pipelines and automated validation of model performance in production 

• Exposure to Agile/Scrum methodology and tools like Azure Boards 

Soft Skills

• Strong problem-solving skills for balancing speed and quality in fast-paced AI development. 
• Ability to communicate technical risks to non-technical stakeholders. 
• Collaborative mindset to work with cross-functional teams (data scientists, ML engineers, DevOps).