Principal AI Evaluation Engineer

As a Principal AI Evaluation Engineer at our company, you will lead the evaluation efforts within our AI-powered SDLC team. Your main responsibilities will include defining and leading the evaluation strategy and roadmap for the core product, building and overseeing evaluation pipelines and guardrails, and analyzing evaluation results to produce actionable reports for engineering and product stakeholders. Additionally, you will be expected to mentor engineers, lead triage and reporting, and ensure that evaluation becomes a cornerstone of release decisions. Key Responsibilities: - Define and lead the evaluation strategy and roadmap for the AI-powered SDLC core product. - Build and oversee evaluation pipelines and guardrails. - Build and maintain evaluation datasets (synthetic and real project data) to benchmark AI behavior. - Analyze evaluation results, identify gaps, and produce clear, actionable reports for engineering and product stakeholders. - Build a culture of innovation and excellence, encouraging continuous improvement and adoption of best practices in AI evaluation and deployment. - Collaborate with cross-functional teams to integrate evaluation insights into development. Qualifications Required: - Strong understanding of software engineering principles and the software development lifecycle (SDLC). - Hands-on experience with test design, test management, observability, and data analysis. - Proficiency in Python (or another scripting language) for automating evaluations. - Familiarity with AI Agent evaluation methods (faithfulness, answer relevancy, contextual accuracy, tool correctness). - Excellent analytical and problem-solving skills. - Strong communication and collaboration abilities, able to work with cross-functional teams and stakeholders.,