MLops Engineer

Role Overview: As a MLOps Engineer at Celebal Technologies, you will be an integral part of the data and AI engineering team. Your primary responsibility will be to leverage your expertise in the Databricks ecosystem to deploy, operationalize, and maintain machine learning models at scale. Working closely with data scientists, data engineers, and business stakeholders, you will play a key role in building robust, automated, and reliable ML pipelines in a production environment. Key Responsibilities: - Manage Databricks Platform: - Utilize Databricks Workspaces, Jobs, Workflows, Unity Catalog, Delta Lake, and MLflow for experiment tracking and model lifecycle management. - Optimize Databricks clusters, compute resources, and workspace permissions. - End-to-End ML Lifecycle (MLOps): - Implement and manage the complete ML lifecycle including model training, versioning, deployment, monitoring, and retraining. - Design and support model deployment strategies such as A/B testing, blue-green deployments, and canary releases. - Programming & Development: - Develop scalable ML and data pipelines using Python (pandas, scikit-learn, PyTorch/TensorFlow), PySpark, and SQL. - Maintain code quality through Git-based version control, reviews, and automated tests. - Cloud & Infrastructure: - Work across cloud environments (AWS/Azure/GCP) to deploy and manage ML infrastructure. - Implement and maintain Infrastructure as Code using Terraform. - Build and manage containerized ML workloads using Docker or Kubernetes. - CI/CD & Automation: - Create and optimize CI/CD pipelines using Jenkins, GitHub Actions, or GitLab CI for ML workflows. - Automate data validation, feature generation, model training, and deployment pipelines. - Monitoring & Observability: - Configure and implement monitoring solutions using Databricks Lakehouse Monitoring for data quality, model performance, model drift, and inference pipelines. - Integrate model explainability tools including SHAP and LIME. - Feature Engineering & Optimization: - Build and manage features with Databricks Feature Store. - Run distributed training and hyperparameter tuning using Optuna, Ray Tune, or similar tools. - Collaboration & Documentation: - Collaborate cross-functionally with data scientists, ML engineers, DevOps teams, and business units. - Create clear, maintainable documentation for pipelines, processes, and systems. Qualifications Required: - Strong experience (3+ years) as a MLOps Engineer or similar role. - Proficiency in working with Databricks ecosystem and Azure cloud. - Hands-on experience with Python, PySpark, SQL, and cloud environments (AWS/Azure/GCP). - Familiarity with CI/CD tools, Infrastructure as Code, and monitoring solutions. - Excellent communication skills and ability to collaborate effectively with cross-functional teams. (Note: The JD does not include any additional details about the company),