Sr Machine Learning Quality Engineer

1. Take ownership of end-to-end quality deliverables for ML/AI products

2. Act as the voice of quality, collaborating with Product and Development teams to define comprehensive test strategy while establishing and adhering to quality milestones, and execute according to plan

3. Build test strategies for AI powered applications like conversational interfaces, Agentic workflows and etc.

4. Evaluate the quality, accuracy, performance, and safety of AI-generated content produced by LLMs and identify weaknesses

5. Create and maintain test automation frameworks

6. Lead development of test datasets (synthetic, autogenerated, edge cases)

7. Collect, analyze, and report quality metrics from test execution and create defects based on results patterns

8. Contribute to refining evaluation criteria and methodologies for model assessment to improve AI evaluation frameworks

9. Support engineering organizations in troubleshooting or addressing issues with applications and dev/test environments

10. Stay up to date with latest developments in LLMs and AI ethics to improve evaluation techniques

1. 5+ years professional experience in software quality engineering with a focus on AI/Machine learning

Hands-on experience with testing machine learning, Generative AI systems and Conversational interfaces, including validation of model predictions and integration testing of end-to-end AI workflows

3. Working knowledge of synthetic data generation techniques and prompt engineering

4. Understanding of core AI principles, including Gen AI, agentic AI, algorithms, models, and ethical considerations

5. Experience working within different automated testing frameworks, including Java, JUnit, Selenium, TestNG and other open-source projects

6. Ability to build testing strategies for AI powered applications such as conversational interfaces like AI Search, Virtual Agent along with Agentic Workflows and etc.

6. Ability to diagnose issues, come up with a strategy for relief, permanent fix and RCA and implement it

7. Ability to implement best practices and coding standards on role specific requirements (e.g. programming languages and tools, network design, functional design, algorithms, patterns, design and development, etc.)

8. Fluency in programming language / scripting in Java and Python

9. Experience with different testing techniques (e.g. performance, unit, integration, automated), their strengths and weakness, and ability to use them to best effect

10. Expert working knowledge and ability to use tools (such as IDE, debugger, build tools, source control, ServiceNow instances, profilers, system administration/Unix tools) to assist with daily tasks

11. Experience with the agile methodology for software development teams

12. Experience driving cross-team technical discussions with an understanding of SDLC concepts