Job
Description
As an AI QASpecialist within Acentra Healths AI & Analytics team, you will ensure theaccuracy, reliability, and validity of our AI, ML, and Generative AI (GenAI)products and services. Leveraging your expertise in quality assurancemethodologies, you will perform Functional, UI, and API testing to effectivelyvalidate products such as AI chatbots, ML/GenAI-generated clinical documents,microservices, and other software solutions. Your role will tackle uniquechallenges in GenAI environments, including output quality assurance andhallucination detection, ensuring stringent standards are consistently met.Aligned with this, you will support and contribute to our maturing AIInter-Rater Reliability (AI-IRR) process and services. Collaboratingclosely with data scientists, software developers, engineers, and stakeholders,you will design and execute comprehensive test plans and strategies specific toour AI and GenAI platforms. Responsibilities include creating and refiningdetailed test cases, verifying prompt-response accuracy, and ensuring theethical and responsible use of AI outputs, ultimately contributing to improvedpatient outcomes. KeyResponsibilities Test Planning & Execution: Develop andimplement test strategies, detailed test plans, and test cases tailored to AI,ML, and GenAI solutions. Quality Assurance & UAT: Conduct thoroughFunctional, UI, and API testing, including GenAI output quality, hallucinationdetection, and user acceptance testing to ensure optimal applicationperformance. Collaboration: Work closely with data scientists,software developers, engineers, and project managers to validate algorithms andintegrate QA best practices into development processes. Automation & Optimization: Implementautomation testing frameworks to optimize testing efforts, including automatedGenAI evaluation workflows. Data Integrity & Bias Detection: Verifytraining data integrity, perform bias detection, and conduct prompt-responsevalidations for AI and GenAI model training. AI Inter-Rater Reliability: Review and verifyAI-generated outputs for consistency and accuracy. Collaborate with AI/MLengineers to enhance automated evaluation systems and pipelines. Documentation & Reporting: Document testresults, identify defects, collaborate to resolve issues, and preparecomprehensive reports with actionable insights for continuous improvement. Compliance & Best Practices: Ensure adherenceto industry standards, regulatory requirements, and Acentra Health standardsaround ethical and safe AI usage. Continuous Improvement: Stay current withemerging AI testing methodologies and tools; contribute to evolving QAprocesses and strategies. Whoyou are: Education & Experience: Bachelors degree in computer science, computer engineering, software engineering, or a related area, or equivalent experience, desired. Technical Skills: Strong experience with QA tools andframeworks like PyTest, Selenium, or JUnit, including Gen AI evaluationframeworks like RAGAS. Understanding of AI, ML, and GenAI concepts,algorithms, and output validation techniques. Familiarity with data annotation tools and qualitycontrol processes for GenAI-specific tasks. Solid knowledge of Python, with experience in Gen AIlibraries like LangChain and LlamaIndex Experience with GenAI model evaluation metrics such asperplexity and BLEU. Solid knowledge of software development life cycle(SDLC) and agile methodologies. Preferred Qualifications Experience in healthcare technology or data analyticsenvironments. Familiarity with regulatory and ethical standardsrelated to AI and healthcare solutions. Soft Skills: Excellent analytical and problem-solving skills withkeen attention to detail. Strong communication and collaboration skills, withthe ability to explain complex technical issues to diverse audiences. Ability to work effectively in a fast-paced,cross-functional team environment.
