Role Overview: Join our mission to ensure AI agents are safe, reliable, and trustworthy for enterprise deployment. You'll focus on developing tests that evaluate agent behavior, identify edge cases, and ensure compliance with safety standards. Key Responsibilities: Design and execute safety testing protocols for AI agents Develop adversarial testing strategies to identify agent vulnerabilities Create test cases for hallucination detection, bias evaluation, and toxic output prevention Build automated monitoring for agent drift and performance degradation Test agent behavior under resource constraints and failure scenarios Evaluate agent compliance with industry-specific regulations Document safety issues and work with developers on mitigation strategies Required Qualifications: 5+ years in software testing with focus on security or safety-critical systems Experience with AI/ML model testing and evaluation Strong analytical and problem-solving skills Knowledge of AI ethics and responsible AI principles Experience with security testing tools and methodologies Excellent documentation and communication skills Preferred Qualifications: Background in cybersecurity or safety engineering Experience with red team/blue team exercises Knowledge of formal verification methods Familiarity with AI incident databases and failure analysis
Company Description AI Guru builds innovative AI augmentation tools that significantly enhance the capabilities of elite professionals. Our mission is to democratize the use of AI superpowers, ensuring every ambitious professional can benefit from the advanced capabilities traditionally reserved for top consulting firms and Fortune 500 companies. Headquartered by industry veterans who have designed leading AI systems at esteemed organizations like Bloomberg, AWS, and Cerebras, AI Guru's tools are trusted by over 20,000 professionals globally, driving substantial career advancements and operational efficiencies. Role Description Design and implement the application layer that connects large language models (LLMs) to real-world data pipelines. You will build and maintain the orchestration logic that retrieves relevant context, feeds it to LLMs, and returns reliable, structured outputs for production systems. Key Responsibilities Architect and maintain the end-to-end LLM orchestration pipeline (retrieval → prompt construction → model call → post-processing). Create reusable prompt templates and dynamic context builders for diverse data sources. Develop deterministic post-processing and validation layers (schema enforcement, range/regex checks). Integrate LLM outputs into backend APIs and user-facing applications. Monitor and optimize LLM performance for latency, accuracy, and cost. Collaborate with backend, data, and QA teams to improve accuracy and robustness. Implement safeguards such as rate limiting, fallback strategies, and prompt versioning. Required Skills & Experience Strong programming skills in Python or TypeScript/Node.js for production services. Hands-on experience with LLM frameworks (e.g., LangChain, LlamaIndex , or similar orchestration tools). Expertise in prompt engineering and structured output handling (e.g., JSON schemas). Familiarity with vector databases (Pinecone, Weaviate, pgvector, etc.) and retrieval strategies. Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes), and cloud deployment (AWS/GCP/Azure). Strong testing habits for data- and prompt-driven applications. Nice to Have Experience with unstructured data (documents, email, audio, etc.) or information extraction. Background in evaluation metrics for retrieval and generation (recall@k, F1, nDCG). Understanding of event-driven architectures and message queues (Kafka/SQS). ====== No head hunters please