Senior LLM Engineer (Prompting & Evaluation)

Job Description

Key Responsibilities ● Prompt Design: Craft and continuously improve prompts for OpenAI, Anthropic and other foundation models using few-shot, chain-of-thought, context-tuning and other techniques for text-analytics and reasoning use cases. ● Evals & Experiments – Develop an evals strategy and build automated eval suites (precision, recall, cost, downstream impact) that run in CI and in production. Build and maintain data sets. ● Prompt Library Management – Stand up a versioned prompt repo, integrate context-injection patterns, and automate rollout/rollback. ● Drift & Performance Monitoring – Detect and guard against context or model shifts ● Context Injection: Use Retrieval augmented generation (RAG) and vector search to inject context information to generate contextually accurate and grounded model responses. Use MCP to manage the way context is assembled, updated and passed to LLM Qualifications ● 5+ Years of AI software experience. Experience with pre-LLM AI tech counts. ● Proven success using LLMs for text analytics or reasoning (not chatbots, style transfer, or safety tuning alone). ● Mastery of prompt-engineering techniques (few-shot, CoT, context, etc.) and hands-on with OpenAI / Anthropic APIs. ● Experience with fine-tuning or adapting foundation models via RLHF, instruction tuning, or domain-specific datasets. ● Proven experience using LLMs via APIs or local deployment (including OpenAI, Claude, Llama) ● Experience building evaluation pipelines for production scale implementations using toolkits like AWS bedrock. ● Strong eval chops—comfortable building custom benchmarks or using tools like OpenAI Evals, Braintrust, etc. ● Solid Python, plus the engineering rigor to wire up automated eval pipelines, data viewers, and model-selection logic. ● Strong written and verbal communicator who can explain trade-offs to both engineers and product leaders Job Type: Full-time Pay: Up to ₹1,500,000.00 per year Benefits: Provident Fund Schedule: Morning shift Supplemental Pay: Performance bonus Experience: LLM: 4 years (Preferred) LLMs for text analytics or reasoning: 3 years (Preferred) AI: 2 years (Preferred) Work Location: In person