DataZymes - Databricks Platform Engineer - ETL

Role Overview :

Seasoned Databricks Senior Platform Engineer who can implement scalable, end-to-end data solutions on Databricks over AWS. This role is pivotal in building the foundational infrastructure for data products, ensuring data quality, and enabling machine learning capabilities

Key Responsibilities :

- Design and implement end-to-end ETL pipelines in Databricks, sourcing data from Amazon S3 and Amazon RDS and other sources.

- Design and deploy Databricks workspaces and clusters optimized for performance, scalability, and cost-efficiency.

- Set up data quality frameworks and anomaly detection mechanisms to ensure reliability and trust in data products.

- Enable machine learning capabilities within Databricks by configuring necessary services, libraries, and integrations.

- Collaborate with data scientists, engineers, and DevOps teams to ensure seamless platform operations.

- Automate deployment and monitoring using CI/CD pipelines, GitHub

- Document architecture, workflows, and operational procedures for platform governance.

Requirements :

Core Platform & Cloud :

- 5+ years of experience in platform engineering roles

- Strong expertise in Databricks on AWS

- Proficient with Amazon S3, Amazon RDS, and IAM roles/policies

- Experience with Delta Lake, Structured Streaming, and Databricks Workflows

ETL & Data Quality :

- Hands-on experience building ETL pipelines using Spark

- Familiarity with data validation, profiling, and anomaly detection frameworks

- Knowledge of tools like Great Expectations or custom validation scripts

ML Platform Enablement :

- Experience setting up ML infrastructure including :

- MLflow for experiment tracking and model registry

- Feature Store for reusable feature pipelines

- Databricks Runtime for ML

- Integration with AWS SageMaker or custom ML libraries

- Understanding of model lifecycle management, though model development is out of scope

DevOps & Automation :

- Proficiency with GitHub, CI/CD pipelines, and Terraform or CloudFormation

Preferred Qualifications :

- Experience working with healthcare data

- Exposure to data mesh or data product architecture principles