Job
Description
Key Responsibilities:Assess and understand existing data platforms e.g., Hadoop, Spark, onpremises systems, or other cloud data warehouses Design and implement end-to-end migration strategies to Databricks. Migrate data pipelines, notebooks, models, and jobs to Databricks. Optimize performance and cost-efficiency during and after the migration. Develop and maintain CI/CD pipelines for Databricks-based projects. Collaborate with data engineers, analysts, and business stakeholders to ensure smooth transition and minimal disruption. Document migration processes, architecture, and configurations. Required Skills & Qualifications 5+ years of experience in data engineering or related roles. 2+ years of hands-on experience with Databricks. Experience migrating from systems like Hadoop, Spark, or legacy ETL tools to Databricks. Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake. Familiarity with cloud platforms (Azure, AWS, or GCP) especially Azure Databricks or AWS Databricks. Experience with CI/CD, DevOps, and version control tools (Git, Azure DevOps, etc.). Strong problem-solving skills and ability to work independently. Excellent communication and documentation skills. Preferred Qualifications Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional). Experience with MLflow, Unity Catalog, or data governance frameworks. Familiarity with Terraform or Infrastructure as Code (IaC) for Databricks setup. Show more Show less
