ML Research Engineer

JD: Machine Learning Research Engineer

Location: Hyderabad

About Us

Deccan AI, founded by IIT Bombay and IIM Ahmedabad alumni, specializes in LLM model development and AI-first scaled operations. Based in SF and Hyderabad, our mission is to create AI for Good, driving innovation with positive societal impact

About the Role.

We are seeking a Machine Learning Research Engineer focused on Data Quality to ensure our model training data meets the highest standards of reliability, relevance, and safety. This role plays a pivotal part in the ML lifecycle — from automated QA of training data to developing evaluation strategies and leading rater workflows — ensuring that the data shipped aligns closely with client expectations and model performance objective

s.You will be at the intersection of engineering, research, and client success, acting as the final quality gatekeeper for datasets powering LLM fine-tuning, reward modeling, and evaluatio

n.Key Responsibilities

Dataset Quality Automation

• Automate quality assurance pipelines for SFT transcripts and RLHF preference pairs.
• Implement schema validation, semantic overlap checks, and embedding-based deduplicatiion
• Integrate filters for safety, toxicity, and reward-signal balance in datasets

Training & Benchmarking

• Execute proxy fine-tuning (LoRA/QLoRA) on open-source LLMs using QA-approved datasets.
• Train lightweight reward models and track performance via public/internal benchmarks and calibration metrics.

LLM Evaluation

• Orchestrate human and LLM-as-judge evaluations, including generation of critiques and sco
• ring.Design evaluation rubrics focused on consistency, helpfulness, and alignment with reward models.
• Calculate and interpret statistical measures like binomial confidence intervals for evaluation scores.

Annotation & Rater Management

• Build a continuous feedback loop with annotation teams, resolve disputes, and maintain high annotation quality.
• Manage human evaluation workflows to maximize consistency and throughput.

Research & Tooling

• Prototype new signal-to-noise metrics (e.g., reward model entropy, preference flip rate).
• Package tooling into reproducible notebooks and integrate into CI pipelines (Airflow/Dagster).

End Value to the Company

as the client-end MLE

Required Skills & Qualifications

• Strong understanding of LLM training and evaluation pipelines (SFT, RLHF, reward modeling).
• Experience with model performance diagnostics, identifying root causes in model behavior (e.g., data flaws, prompt issues).

• S

  killed in prompt en

  g

  ineering, dataset schema design, and annotation guideline development.
• Proficient in Python, with experience using PyTorch, Hugging Face Transformers, FastAPI.
• Comfortable building evaluation frameworks, including leaderboards and domain-specific
• test sets.
• Familiarity with model evaluation metrics, clustering techniques, embedding models, and data drift detection.
• Strong communication skills, especially in translating technical findings into actionable client
• insights.Self-start

  er with a consultati

  ve mindset who can operate across technical and business domains.