Gen AI Validation Engineer

About the Role:
The Team:
A part of Ratings Technology, the Ratings QA team leads the IT organization in the delivery of trusted quality to our users. From a foundation of disciplined processes, QA seeks out innovative testing solutions, reliable & transparent test execution, and aspires towards continuous improvement in achieving top quality.

In close partnership with Product Management and Product Engineering, QA team members help drive quality accountability, and work to ensure our end users receive reliable, stable, defect free products, which help confidently execute our business and elevate our products.
Responsibilities:

• Responsible for certifying applications that integrate Generative AI models & workflows
• Validate the functionalities of the In-House Generative AI platform that serves models to internal users.
• Ensure high-quality standards and performance of applications without involvement in model evaluations.
• Develop automation/performance scripts that meet organization standards and build reliable, reusable and maintainable automated regression suites & test harness
• Leverage tools and frameworks to build automation/performance scripts with quality code to simplify testing scenarios
• Work in an Agile environment and adhere to all QA best practices.
• Work in partnership with Scrum team members, including engineering and product management, to deliver business functionality on-time, with required quality, and that meets the acceptance criteria
• Write test cases and scenarios from the requirements/designs; manage bug tracking.
• Trigger, monitor and control the execution of automated test suites, identifying issues and potential gaps.
• Support QA governance of test processes, data, and execution metrics to help drive improve the performance of team.
• Perform risk analysis for features/projects in your ownership, provide mitigation and escalate appropriately.
• Collaborate effectively with technical and non-technical stakeholders to analyze, debug and resolve any issues.

• 6 to 10 years experience in certifying complex business applications workflows
• 1-2 years of experience exposure in working with Generative AI applications
• Exposure or awareness of LLM model evaluations
• Practitioner level understanding of prompt engineering
• Strong hands-on coding/scripting experience in automation testing & engineering programming using Python/Java languages.
• Experience in design and development of automated tests using automation tool (e.g., Python, Selenium) across the application layers (UI/Service/Data layers) and root cause analysis.
• Experience of working with REST services
• Exposure to distributed source control systems such as Git
• Excellent problem solving, analytical and technical troubleshooting skills.
• Experience of testing in CI/CD/CT, Azure DevOps model.
• In depth usage and creation of automation framework for testing JSON, Rest/SOAP based web services
• Bachelor's or higher qualification in Computer Science, Information Systems or equivalent is preferred.
• Knowledge of testing in continuous integration/DevOps models is a plus.
• Understanding of Cloud technologies like AWS/Azure and micro-services, containers integration testing will be added advantage.