QA Engineer III

As a member of our Agentic AI Engineering team, we seek individuals passionate about advancing the quality and reliability of AI systems through rigorous evaluation, experimentation, and tooling. You will play a key role in assessing the performance, safety, and alignment of AI models, leveraging platforms like OpenAI and LangSmith. Our team thrives in a culture of curiosity, continuous learning, and collaborative innovation, where your contributions directly impact the trustworthiness of AI solutions used globally.

• • Design and execute evaluation strategies for LLM-based systems using LangSmith, OpenAI tools, and custom frameworks.
  • Collaborate with product managers, researchers, and engineers to define evaluation metrics aligned with business and ethical goals.
  • Develop automated pipelines for regression testing, prompt evaluation, and model comparison.
  • Analyze model outputs for correctness, bias, safety, and hallucination using both qualitative and quantitative methods.
  • Build reusable test harnesses and datasets for benchmarking AI capabilities across tasks and domains.
  • Contribute to the development of internal tools for prompt engineering, evaluation visualization, and feedback loops.
  • Act as a quality advocate for AI-driven features across the product lifecycle.
  • Mentor junior team members in AI evaluation best practices and tooling.
  • Stay current with advancements in LLM evaluation, interpretability, and responsible AI.

• Bachelor's or master's degree in computer science or a related technical field.
• Minimum 6-8 years of experience into QA and Automation
• Basic level understanding of software development lifecycle and quality best practices and tools
• Good knowledge on Automation Framework Development
• Proficient in the use of various automation frameworks like Playwright (preferred), Protractor, Rest Assured, karate etc.
• Hands-on experience with OpenAI APIs (e.g., GPT-4, function calling, Assistants API).
• Familiarity with OpenAI Evals and LangSmith for prompt orchestration and evaluation.
• Strong Python skills, especially in building test automation and data analysis scripts.
• Experience with prompt engineering, LLM fine-tuning, or RAG pipelines is a plus.
• Proficiency in using tools like Jupyter, Pandas, and/or SQL for evaluation analytics.
• Understanding of ethical AI principles, model alignment, and safety concerns.
• Experience with version control (GitHub), CI/CD pipelines, and test management tools (e.g., JIRA, Azure Test Plans).
• Excellent communication and collaboration skills, with the ability to explain complex AI behaviors to diverse stakeholders.

Bachelor's or master's degree in computer science or a related technical field.