Skill
10+ years in Quality Engineering, Automation Testing, with 3+ years in Gen AI, Python, LLM, RAG
Location :
Trivandrum, Bangalore, Chennai, Kochi, Hyderabad, Pune, Noida, gurgaon, KolkataExp : 10 Years +QE Architect – AI/LLM Systems to architect, define, and drive the overall Quality Engineering strategy and frameworks for advanced AI products. As a QE Architect, you will design scalable QE architectures, establish AI quality frameworks, create evaluation pipelines, lead automation strategy, and ensure that all AI components deliver reliable, safe, high-quality user experiences. This role requires thought leadership, deep technical expertise, and the ability to work at the intersection of AI/ML, software engineering, and quality. Key Responsibilities: 1. QE Architecture & Strategy Define and own the quality architecture for all AI/LLM initiatives across the organization. Develop enterprise-level frameworks and reusable QE components for: Conversation apps and chatbots Knowledge-management bots and RAG systems Image search AI Semantic/vector-based text search Multimodal and generative AI systems Establish scalable testing pipelines for model evaluation, data validation, and automation. 2. AI/LLM Evaluation Frameworks Architect comprehensive evaluation systems for: Prompt testing & scenario-based testing LLM output quality, safety, bias, and consistency Hallucination detection frameworks RAG correctness, grounding accuracy, and knowledge integrity Search ranking metrics Create automated scorecards and continuous evaluation dashboards. 3. Automation & Infrastructure Design automation frameworks for LLM APIs, chat agents, multimodal pipelines, and vector search services. Architect model-regression detection systems using: Golden datasets Synthetic-data generation LLM-as-a-judge methodologies Self-evaluation and multi-agent evaluation Integrate automated test harnesses into CI/CD and LLMOps pipelines. 4. Data Quality & Test Data Strategy Define enterprise-wide test data management for AI, including: Ground-truth datasets Benchmarking datasets Adversarial inputs Safety and edge-case 5. Architecture Reviews & Cross-team Leadership Provide architectural guidance to ML engineers, data engineers, and software teams on testability and observability. Review architecture of model pipelines, chatflows, orchestration components, and search systems. Drive AI quality gates across experimentation, pre-production, and production rollout cycles. 6. Quality Governance & Best Practices Create enterprise standards for: AI testing taxonomies Privacy, safety, and compliance validation Defect classification for LLM issues Reliability, latency, and scalability benchmarks Lead internal adoption of AI/ML quality engineering practices. Core Technical Competency 10+ years in Quality Engineering, with 3+ years in AI/ML/LLM systems. Deep understanding of: LLMs, NLP models, embeddings, vector databases Chatbot platforms (Dialogflow, Rasa, Botpress, Lex, etc.) RAG pipelines and knowledge-management systems Image search / multimodal AI architectures Strong programming expertise in Python, Java, or TypeScript with ML/NLP libraries. Experience building CI/CD-integrated AI test automation frameworks. Knowledge of metrics such as perplexity, grounding score, factuality, CER/WER, MRR, NDCG, BLEU/ROUGE, model drift metrics, etc. Proven experience architecting large-scale quality engineering systems. Expert in test automation design patterns, quality frameworks, and QE best practices. Experience with non-deterministic testing, probabilistic evaluation, and AI quality challenges.Skills
AI/ML/LLM system, Python, Java or TypeScript,CI/CD,QE Architecture
