Job
Description
As a Quality Engineer at Mindtickle, you will be responsible for owning the end-to-end qualification lifecycle for AI/LLM systems. Your key responsibilities will include: - Designing and implementing scalable automated test suites across unit, integration, regression, and system levels. - Building and enhancing frameworks to test, evaluate, and continuously improve complex AI and LLM workflows. - Leading the design and automation of LLM-powered features, such as prompt pipelines, RAG workflows, and AI-assisted developer tools. - Developing evaluation pipelines to measure factual accuracy, hallucination rates, bias, robustness, and overall model reliability. - Defining and enforcing metrics-driven quality gates and experiment tracking workflows to ensure consistent, data-informed releases. - Collaborating with agile engineering teams, participating in design discussions, code reviews, and architecture decisions to drive testability and prevent defects early (shift left). - Developing monitoring and alerting systems to track LLM production quality, safety, and performance in real time. - Conducting robustness, safety, and adversarial testing to validate AI behavior under edge cases and stress scenarios. - Continuously improving frameworks, tools, and processes for LLM reliability, safety, and reproducibility. - Mentoring junior engineers in AI testing, automation, and quality best practices. - Measuring and improving Developer Experience (DevEx) through tools, feedback loops, and automation. - Championing quality engineering practices across the organization, ensuring delivery meets business goals, user experience, cost of operations, etc. In order to excel in this role at Mindtickle, you should have experience with LLM testing & evaluation tools such as MaximAI, OpenAI Evals, TruLens, Promptfoo, and LangSmith. Additionally, you should have expertise in building LLM-powered apps, CI/CD design for application and LLM testing, API, performance, and system testing, as well as Git, Docker, and cloud platforms like AWS, GCP, or Azure. You should also be knowledgeable in bias, fairness, hallucination detection, and AI safety testing, with a passion for mentorship and cross-functional leadership. Preferred qualifications for this position include: - A Bachelors or Masters degree in Computer Science, Engineering, or equivalent. - 4+ years of experience in software development, SDET, or QA automation. - Proficiency in GoLang, Java, or Python. - Proven experience in building test automation frameworks. - Ability to design CI/CD pipelines with automated regression and evaluation testing. - Hands-on exposure to LLMs, GenAI applications. - 2+ years of hands-on experience with LLM APIs and frameworks such as OpenAI, Anthropic, Hugging Face. - Proficiency in prompt engineering, embeddings, RAG, and LLM evaluation metrics. - Strong analytical, leadership, and teamwork skills. - Excellent communication and collaboration abilities across teams. We look forward to hearing from you if you meet these qualifications and are excited about the opportunity to contribute to our innovative team at Mindtickle. As a Quality Engineer at Mindtickle, you will be responsible for owning the end-to-end qualification lifecycle for AI/LLM systems. Your key responsibilities will include: - Designing and implementing scalable automated test suites across unit, integration, regression, and system levels. - Building and enhancing frameworks to test, evaluate, and continuously improve complex AI and LLM workflows. - Leading the design and automation of LLM-powered features, such as prompt pipelines, RAG workflows, and AI-assisted developer tools. - Developing evaluation pipelines to measure factual accuracy, hallucination rates, bias, robustness, and overall model reliability. - Defining and enforcing metrics-driven quality gates and experiment tracking workflows to ensure consistent, data-informed releases. - Collaborating with agile engineering teams, participating in design discussions, code reviews, and architecture decisions to drive testability and prevent defects early (shift left). - Developing monitoring and alerting systems to track LLM production quality, safety, and performance in real time. - Conducting robustness, safety, and adversarial testing to validate AI behavior under edge cases and stress scenarios. - Continuously improving frameworks, tools, and processes for LLM reliability, safety, and reproducibility. - Mentoring junior engineers in AI testing, automation, and quality best practices. - Measuring and improving Developer Experience (DevEx) through tools, feedback loops, and automation. - Championing quality engineering practices across the organization, ensuring delivery meets business goals, user experience, cost of operations, etc. In order to excel in this role at Mindtickle, you should have experience with LLM testing & evaluation tools such as MaximAI, OpenAI Evals, TruLens, Promptfoo, and LangSmith. Additionally,
