Quality Analyst - AI ML, Python, Cloud, Selenium

Caring. Connecting. Growing together.

We are seeking a Quality Analyst with an AI Scientist persona focused on implementing and testing an AI Multiagentic Framework that automates manual claims adjudication using AI agents. This role blends traditional Quality Engineering, AI/ML expertise, and cloud-based pipeline quality assurance to ensure robust, compliant, and scalable AI solutions.

Primary Responsibilities:

• Validate data ingestion, feature engineering, and model training pipelines for accuracy and consistency
• Implement unit, integration, and end-to-end tests for AI workflows
• Verify model outputs against ground truth datasets and compliance rules
• Test orchestration of multiple AI agents for claims adjudication workflows
• Ensure seamless integration and testing of AI pipelines deployed on AWS, Google Cloud (GCP), and Azure
• Validate AI services and frameworks like AWS SageMaker, AWS Bedrock, Google Vertex AI, TensorFlow on GCP, and Azure Databricks/Data Factory
• Design and execute functional, regression, and performance test plans
• Automate test cases using industry-standard frameworks
• Drive automation in pipeline validation and regression testing
• Ensure seamless integration with enterprise systems and APIs
• Collaborate with Business & Data SMEs and engineers to refine models and workflows
• Drive improvements in automation coverage and decision accuracy
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Knowledge of AI/ML & Pipeline Quality:
  • Data Validation: Schema checks, data drift detection, anomaly detection
  • Gen AI Tools/Models - Gen AI Foundation Models , Prompt Engineering , ChatGPT , Llama , Anthropic Claude , Gemini , Copilot
  • Model Validation: Accuracy, bias, fairness, explainability testing
  • ML Ops: CI/CD for AI pipelines, automated deployment, rollback strategies
  • Monitoring: Metrics for model performance, pipeline health, anomaly alerts
• Knowledge of Cloud & AI Frameworks: AWS SageMaker, Bedrock, Lambda GCP Vertex AI, TensorFlow Extended Azure Databricks, Data Factory
• Knowledge of Programming Languages: Python, SQL, T-SQL, HTML
• Knowledge of Languages: Python (AI workflows), automation scripts
• Knowledge of Version Control: GitHub for code management
• Knowledge of Data & Integration: T-SQL for data validation, Kafka, Spark for distributed processing
• Knowledge of Testing Frameworks:
  • UI/Frontend - Selenium, Cypress
  • Performance and API - JMeter, Postman, REST Assured
  • Unit Testing - PyTest, JUnit, NUnit
• Knowledge of Version Control: Git and GitHub
• Knowledge of Agile Practices: Experience in Agile/Scrum environments
• Understanding performance optimization for AI workflows
• Experience with Kafka for event streaming and Spark for distributed processing

Preferred Qualifications:

• Experience with RAG (Retrieval-Augmented Generation) and AI explainability tools
• Knowledge of cloud security best practices for AI deployments
• Familiarity with LangGraph, AutoGen, CrewAI , or similar agentic AI frameworks
• Exposure to healthcare claims adjudication and compliance standards (HIPAA, CMS)