Quality Assurance Specialist

Job Description

Job Title: AI QA Specialist Location: Chennai/Bangalore/Hyderabad - Hybrid About the Job: As an AI QA Specialist within Acentra Health’s AI & Analytics team, you will ensure the accuracy, reliability, and validity of our AI, ML, and Generative AI (GenAI) products and services. Leveraging your expertise in quality assurance methodologies, you will perform Functional, UI, and API testing to effectively validate products such as AI chatbots, ML/GenAI-generated clinical documents, microservices, and other software solutions. Your role will tackle unique challenges in GenAI environments, including output quality assurance and hallucination detection, ensuring stringent standards are consistently met. Aligned with this, you will support and contribute to our maturing AI Inter-Rater Reliability (AI-IRR) process and services. Collaborating closely with data scientists, software developers, engineers, and stakeholders, you will design and execute comprehensive test plans and strategies specific to our AI and GenAI platforms. Responsibilities include creating and refining detailed test cases, verifying prompt-response accuracy, and ensuring the ethical and responsible use of AI outputs, ultimately contributing to improved patient outcomes. Key Responsibilities Test Planning & Execution: Develop and implement test strategies, detailed test plans, and test cases tailored to AI, ML, and GenAI solutions. Quality Assurance & UAT: Conduct thorough Functional, UI, and API testing, including GenAI output quality, hallucination detection, and user acceptance testing to ensure optimal application performance. Collaboration: Work closely with data scientists, software developers, engineers, and project managers to validate algorithms and integrate QA best practices into development processes. Automation & Optimization: Implement automation testing frameworks to optimize testing efforts, including automated GenAI evaluation workflows. Data Integrity & Bias Detection: Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training. AI Inter-Rater Reliability: Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines. Documentation & Reporting: Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement. Compliance & Best Practices: Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage. Continuous Improvement: Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies. Who you are: Education & Experience: Bachelor’s degree in computer science, computer engineering, software engineering, or a related area, or equivalent experience, desired. Technical Skills: Strong experience with QA tools and frameworks like PyTest, Selenium, or JUnit, including Gen AI evaluation frameworks like RAGAS. Understanding of AI, ML, and GenAI concepts, algorithms, and output validation techniques. Familiarity with data annotation tools and quality control processes for GenAI-specific tasks. Solid knowledge of Python, with experience in Gen AI libraries like LangChain and LlamaIndex Experience with GenAI model evaluation metrics such as perplexity and BLEU. Solid knowledge of software development life cycle (SDLC) and agile methodologies. Preferred Qualifications Experience in healthcare technology or data analytics environments. Familiarity with regulatory and ethical standards related to AI and healthcare solutions. Soft Skills: Excellent analytical and problem-solving skills with keen attention to detail. Strong communication and collaboration skills, with the ability to explain complex technical issues to diverse audiences. Ability to work effectively in a fast-paced, cross-functional team environment. Show more Show less