RAG & Agentic AI Evaluation Engineer

About Us:

Soul AI is a pioneering company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in delivering high-quality human-curated data, AI-first scaled operations services, and more. Based in SF and Hyderabad, we are a young, fast-moving team on a mission to build AI for Good, driving innovation and positive societal impact.

About the Role:

RAG & Agentic AI Evaluation Engineer

evaluate agent behavior

Responsibilities:

• Annotate model responses, reasoning steps, tool usage, and agent actions
• Evaluate RAG output quality: relevance, accuracy, grounding, hallucinations
• Review agent workflows and end-to-end task execution
• Identify incorrect reasoning, missing retrievals, or flawed tool calls
• Provide structured feedback to improve agent behavior and RAG performance
• Validate retrieved documents, sources, and context relevance
• Review multi-hop reasoning chains for correctness and completeness
• Tag error types such as bias, hallucination, logical gaps, retrieval mismatch
• Follow detailed annotation guidelines to ensure consistent evaluations
• Document insights, issues, and improvement suggestions for AI research teams
• Collaborate with model developers to refine prompts, retrieval logic, and agent strategies

Skills & Experience Required:

• 1–4 years of experience working on RAG, or other agentic models.
• Understanding of LLMs, RAG pipelines, embeddings, and retrieval workflows
• Ability to judge correctness of model outputs and identify subtle issues
• Familiarity with agentic workflows (tool use, multi-step tasks, reasoning traces)
• Experience in documentation, evaluation, or quality review processes
• High attention to detail and ability to follow structured guidelines
• Experience with vector databases, prompt engineering, or LLM tools
• Knowledge of multi-hop reasoning, chain-of-thought, or tool invocation
• Prior experience in annotation or AI evaluation projects

Why Join Us?

• Work on high-impact projects that contribute to building AI for Good.
• Collaborate with top-tier engineers and domain experts from IITs, NITs, and BITS.
• Opportunity to grow in a fast-paced, innovation-driven environment.