AI QA Test Engineer

About the Role

AI QA Test Engineer

Participates in QA activities on project assignments and ensures timely delivery on QA commitments in a fast-paced product development environment.

Review of functional requirements and technical design documentation and creation of test plans, test scenarios and test cases. Execution of test plans and certification of application releases

Key Responsibilities:

AI/ML Testing

• Validate ML models, data pipelines, and inference APIs for accuracy, fairness, and robustness
• Evaluate model behavior across diverse data sets, edge cases, and unexpected inputs
• Monitor model drift, performance degradation, and bias over time
• Conduct prompt testing, hallucination detection, and safety validation for LLM-based systems
• Experience of testing implementation of AI/ML into application
• Manual Testing with automation exposure
• API testing with SOAP UI/ Postman
• Agile methodology experience
• Test & Defect management with JIRA/ Azure DevOps (ADO)
• DB Testing Experience  - Good to Have                                                                              
• Tools - Deepeval/ Langfuse/ Pytest

Quality Engineering & Automation

• Build and maintain automated test suites for AI features
• Develop test scripts for ML workflows using Python, Playwright, and CI/CD tools
• Design and execute manual and automated tests for functional, performance, and security scenarios
• Identify defects, establish test metrics, reporting dashboards, and test traceability

Data Quality & Responsible AI

• Validate training and inference data quality, labeling consistency, and data governance
• Work with data science and product teams to implement responsible AI testing guidelines
• Document and escalate risks related to bias, privacy, explainability, and compliance

Cross-Functional Collaboration

• Partner with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria
• Participate in design reviews, model evaluations, and release readiness checkpoints
• Contribute to QA strategy and Test Plans

Required Qualifications

• Bachelor’s degree in computer science, Engineering, or related field (or equivalent experience)
• 4+ years of experience in software QA engineering
• 2+ years working with AI/ML or data-centric systems
• Strong Python programming skills
• Proficiency in automation frameworks (e.g., Playwright, WebdriverIO, Selenium)
• Experience with ML platforms (TensorFlow, PyTorch, Hugging Face, MLflow, etc.)
• Hands-on experience testing APIs, data pipelines, and distributed systems
• Exposure to RAG pipelines and agent frameworks
• Familiarity with prompt testing and LLM evaluation techniques
• Experience with model evaluation metrics (precision, recall, F1, ROC-AUC, BLEU, perplexity, etc.)
• Knowledge of MLOps, CI/CD, and cloud platforms (AWS, Azure, or GCP)
• Familiarity with Responsible AI frameworks (NIST AI RMF, ISO/IEC standards, fairness toolkits)
• Experience testing vector databases and AI search (Pinecone, Elasticsearch, Redis Vector)

Key Skills

• AI/ML testing & analysis
• Automation & scripting
• Data validation & pipelines
• Risk, ethics, & compliance mindset
• Strong problem-solving & debugging skills
• Excellent documentation & communication skills
• Working with distributed teams