Machine Learning Engineer

JD: Machine Learning Engineer

Location:

About Us

Deccan AI, founded by IIT Bombay and IIM Ahmedabad alumni, specializes in LLM model development and AI-first scaled operations. Based in SF and Hyderabad, our mission is to create AI for Good, driving innovation with positive societal impact.

About the Role

We are seeking a Machine Learning Engineer focused on Data Quality to ensure our model training data meets the highest standards of reliability, relevance, and safety. This role plays a pivotal part in the ML lifecycle — from automated QA of training data to developing evaluation strategies and leading rater workflows — ensuring that the data shipped aligns closely with client expectations and model performance objectives.

You will be at the intersection of engineering, research, and client success, acting as the final quality gatekeeper for datasets powering LLM fine-tuning, reward modeling, and evaluation

Key Responsibilities

Dataset Quality Automation

• Automate quality assurance pipelines for SFT transcripts and RLHF preference pairs.
• Implement schema validation, semantic overlap checks, and embedding-based deduplication.
• Integrate filters for safety, toxicity, and reward-signal balance in datasets.

Training & Benchmarking

• Execute proxy fine-tuning (LoRA/QLoRA) on open-source LLMs using QA-approved datasets.
• Train lightweight reward models and track performance via public/internal benchmarks and calibration metrics.

LLM Evaluation

• Orchestrate human and LLM-as-judge evaluations, including generation of critiques and scoring.
• Design evaluation rubrics focused on consistency, helpfulness, and alignment with reward models.
• Calculate and interpret statistical measures like binomial confidence intervals for evaluation scores.

Annotation & Rater Management

• Build a continuous feedback loop with annotation teams, resolve disputes, and maintain high annotation quality.
• Manage human evaluation workflows to maximize consistency and throug

  hput.

Research & Tooling

• Prototype new signal-to-noise metrics (e.g., reward model entropy, preference flip
• rate).Package tooling into reproducible notebooks and integrate into CI pipelines (Airflow/Dagster).

End Value to the Company

as the client-end MLE a

Required Skills & Qualifications

• Strong understanding of LLM training and evaluation pipelines (SFT, RLHF, reward modeling).
• Experience with model performance diagnostics, identifying root causes in model behavior (e.g., data flaws, prompt issues).
• Sk

  i

  lled in prompt engi

  n

  eering, dataset schema design, and annotation guideline development.
• Proficient in Python, with experience using PyTorch, Hugging Face Transformers,FastAPI.
• Comfortable building evaluation frameworks, including leaderboards and domain-specific test sets.
• Familiarity with model evaluation metrics, clustering techniques, embedding models, and data drift detection.
• Strong communication skills, especially in translating technical findings into actionable client insights.
• Self-starter

  with a consultative

  mindset who can operate across technical and business domains.

Nice -to-Have

• Experience with embedding similarity, data deduplication, or dataset filtering for toxicity/safety.
• Prior work in LLM-as-a-Judge systems or human alignment evaluations.
• Familiarity with CI/CD for data workflows and orchestration tools like Airflow or Dagster.