Machine Learning Specialist

KEY ACCOUNTABILITIES

• Design and implement scalable ML infrastructure on Databricks Lakehouse Platform
• Develop and maintain continuous integration and continuous deployment (CI/CD) pipelines for machine learning models using Databricks workflows.
• Create automated testing and validation processes for machine learning models with Databricks MLflow
• Implement and manage model monitoring systems using Databricks Model Registry and monitoring tools
• Collaborate with data scientists, software engineers, and product teams to optimize machine learning workflows on Databricks.
• Develop and maintain reproducible machine learning environments using Databricks Notebooks and clusters.
• Implement advanced feature engineering and management using Databricks Feature Store
• Optimize machine learning model performance using Databricks runtime and optimization techniques.
• Ensure data governance, security, and compliance within the Databricks environment.
• Create and maintain comprehensive documentation for ML infrastructure and processes.
• Working across teams from several suppliers (including IT Provision, system development, business units and Programme management).
• Continuous improvement and transformation initiatives for MLOps / DataOps in RSA

FUNCTIONAL / TECHNICAL SKILLS

• Bachelor’s or master’s degree in computer science, Machine Learning, Data Engineering, or related field
• 3-5 years of experience in ML Ops with demonstrated expertise in Databricks and/or Azure ML
• Advanced proficiency with Databricks Lakehouse Platform
• Strong experience with Databricks MLflow for experiment tracking and model management
• Expert-level programming skills in Python, with advanced knowledge of:
• PySpark
• MLlib
• Delta Lake
• Azure ML SDK
• Deep understanding of Databricks Feature Store and Feature Engineering techniques
• Experience with Databricks workflows and job scheduling
• Proficiency in machine learning frameworks compatible with Databricks and Azure ML (TensorFlow, PyTorch, scikit-learn)
• Strong knowledge of cloud platforms, including Azure Databricks, Azure DevOps, Azure ML
• Strong exposure to Terraform, ARM/BICEP
• Understanding of distributed computing and big data processing techniques
• Experience with Containerisation, WebApps Kubernetes, Cognitive Services and other MLOps tools will be a plus.