Automation Manager

You will be joining a global digital solutions and technology consulting company headquartered in Mumbai, India, with a strong ethical reputation and innovative culture. With an annual revenue exceeding $4.29 billion and a workforce of 86,000 professionals in over 40 countries, the company serves a diverse global client base across various industries. As an Automation Tester (Ai Testing_LLM+Python), you will play a crucial role in evaluating and optimizing Large Language Model (LLM) responses using frameworks like Ragas, LangSmith, and Python-based tools. Your expertise will contribute to enhancing model accuracy, reliability, and performance, ensuring high-quality GenAI solutions. **Key Responsibilities:** - Design and implement test strategies for AI/LLM-based applications to ensure response accuracy and reliability. - Evaluate LLM outputs using frameworks like Ragas, LangSmith, and custom Python-based evaluation pipelines. - Develop automated workflows to measure metrics such as context precision, faithfulness, and answer relevancy. - Write, refine, and optimize prompts to enhance model accuracy and reduce hallucinations. - Analyze and optimize token usage, prompt structure, and context-window efficiency for improved LLM performance. - Conduct performance testing of LLM responses, including measuring inter-token latency, first token time, and response throughput. - Build Python-based scripts, validators, and reporting dashboards for automated model evaluation. - Collaborate with cross-functional teams to enhance model performance and user experience. - Ensure LLM outputs meet functional, business, and compliance requirements. - Create and maintain test documentation including test plans, cases, evaluation reports, and performance benchmarking dashboards. - Integrate LLM testing into CI/CD pipelines for continuous monitoring and regression validation. - Validate test data and evaluation datasets using SQL. **Qualifications Required:** - 6+ years of experience in Automation Testing - Proficiency in Python, basic SQL, and automation principles - Strong expertise in LLM evaluation, Ragas, LangSmith, and AI Testing - Experience in prompt engineering and optimization - Familiarity with performance testing metrics and tools - Mid to Senior level experience in IT Services and IT Consulting (Note: The details about the company have been omitted as they were not explicitly mentioned in the job description.),