Job
Description
Role Overview: As a Quality Assurance (QA) Engineer at OpenText, you will be responsible for validating Generative AI systems, multi-agent workflows, and Retrieval-Augmented Generation (RAG) pipelines developed using frameworks like LangGraph, LangChain, and Crew AI. You will work closely with AI engineers, data scientists, and product owners to ensure the accuracy, reliability, and performance of LLM-powered enterprise applications. Key Responsibilities: - Develop and execute test cases for validating RAG pipelines, LLM integrations, and agentic workflows. - Validate context retrieval accuracy, prompt behavior, and response relevance across different LLM configurations. - Conduct functional, integration, and regression testing for GenAI applications exposed via APIs and microservices. - Test Agent-to-Agent (A2A) & Model Context Protocol (MCP) communication flows for correctness, consistency, and task coordination. - Verify data flow and embedding accuracy between vector databases (Milvus, Weaviate, pgvector, Pinecone). - Build and maintain automated test scripts for evaluating AI pipelines using Python and PyTest. - Integrate AI evaluation tests into CI/CD pipelines (GitLab/Jenkins) to ensure continuous validation of models and workflows. - Support performance testing of AI APIs and RAG retrieval endpoints for latency, accuracy, and throughput. - Assist in creating automated reports summarizing evaluation metrics such as Precision@K, Recall@K, grounding scores, and hallucination rates. - Validate guardrail mechanisms, response filters, and safety constraints to ensure secure and ethical model output. - Use OpenTelemetry (OTEL) and Grafana dashboards to monitor workflow health and identify anomalies. - Participate in bias detection and red teaming exercises to test AI behavior under adversarial conditions. - Document test plans, results, and evaluation methodologies for repeatability and governance audits. - Collaborate with Product and MLOps teams to streamline release readiness and model validation processes. Qualifications Required: - Education: Bachelor's degree in Computer Science, AI/ML, Software Engineering, or related field. - Experience: 4-7 years in Software QA or Test Automation, with at least 2 years exposure to AI/ML or GenAI systems. - Solid hands-on experience with Python and PyTest for automated testing. - Basic understanding of LLMs, RAG architecture, and vector database operations. - Exposure to LangChain, LangGraph, or other agentic AI frameworks. - Familiarity with FastAPI, Flask, or REST API testing tools (Postman, PyTest APIs). - Experience with CI/CD pipelines (GitLab, Jenkins) for test automation. - Working knowledge of containerized environments (Docker, Kubernetes). - Understanding of AI evaluation metrics (Precision@K, Recall@K, grounding, factual accuracy). - Exposure to AI evaluation frameworks like Ragas, TruLens, or OpenAI Evals. - Familiarity with AI observability and telemetry tools (OpenTelemetry, Grafana, Prometheus). - Experience testing LLM-powered chatbots, retrieval systems, or multi-agent applications. - Knowledge of guardrail frameworks (Guardrails.ai, NeMo Guardrails). - Awareness of AI governance principles, data privacy, and ethical AI testing. - Experience with cloud-based AI services (AWS Sagemaker, Azure OpenAI, GCP Vertex AI). - Curious and eager to learn emerging AI technologies. - Detail-oriented with strong problem-solving and analytical skills. - Excellent communicator who can work closely with engineers and product managers. - Passion for quality, reliability, and measurable AI performance. - Proactive mindset with ownership of test planning and execution. Role Overview: As a Quality Assurance (QA) Engineer at OpenText, you will be responsible for validating Generative AI systems, multi-agent workflows, and Retrieval-Augmented Generation (RAG) pipelines developed using frameworks like LangGraph, LangChain, and Crew AI. You will work closely with AI engineers, data scientists, and product owners to ensure the accuracy, reliability, and performance of LLM-powered enterprise applications. Key Responsibilities: - Develop and execute test cases for validating RAG pipelines, LLM integrations, and agentic workflows. - Validate context retrieval accuracy, prompt behavior, and response relevance across different LLM configurations. - Conduct functional, integration, and regression testing for GenAI applications exposed via APIs and microservices. - Test Agent-to-Agent (A2A) & Model Context Protocol (MCP) communication flows for correctness, consistency, and task coordination. - Verify data flow and embedding accuracy between vector databases (Milvus, Weaviate, pgvector, Pinecone). - Build and maintain automated test scripts for evaluating AI pipelines using Python and PyTest. - Integrate AI evaluation tests into CI/CD pipelines (GitLab/Jenkins) to ensure continuous validation of models and workflows. - Support performance testing of AI APIs and RAG retrieval en
