Senior ML Engineer

IndiaAI is currently focused on developing India's next-gen foundational LLMs and is seeking a Senior ML Engineer who is skilled in large-scale pre-training, distributed GPU systems, and data creation pipelines. As a Senior ML Engineer, your primary responsibility will be to work with cutting-edge technologies such as Megatron-LM, NVIDIA NeMo, DeepSpeed, PyTorch Distributed, and SLURM to train models exceeding 7B parameters on multi-node GPU clusters. Key Responsibilities: - Build and optimize LLM pre-training pipelines for models exceeding 7B parameters. - Implement distributed training using PyTorch Distributed, DeepSpeed (ZeRO/FSDP), Megatron-LM, and NVIDIA NeMo. - Manage multi-node GPU jobs through SLURM and enhance NCCL communication efficiency. - Take the lead in large-scale data tasks including creation, cleaning, deduplication, tokenization, and sharding for multilingual datasets with a particular focus on Indian languages. - Develop high-throughput dataloaders, monitoring dashboards, and training workflows. - Collaborate with infrastructure teams to enhance GPU utilization, networking, and storage systems. Qualifications Required: - At least 5 years of experience in ML Engineering and DL Systems. - Previous experience in training large transformer models, ideally exceeding 7B parameters. - Proficiency in NVIDIA NeMo, Megatron-LM, DeepSpeed, and PyTorch Distributed. - Hands-on experience with SLURM and multi-node GPU clusters like A100/H100. - Familiarity with transformer internals including attention mechanisms, RoPE, FlashAttention, and parallelism. - Experience in data pipelines involving cleaning, dataset assembly, and tokenization. Additional Details: - Bonus Skills: Experience with Indic-language data, MoE training, kernel-level optimization (Triton/CUDA), and open-source contributions to frameworks like Megatron, NeMo, DeepSpeed, and PyTorch.,