Newscape - Senior Databricks Engineer - Azure Data Services

Description

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, and Spark SQL.
• Work with Databricks components such as Workspace, Jobs, Delta Live Tables (DLT), Repos, and Unity Catalog.
• Implement Delta Lake and Lakehouse (Medallion) architectures for optimized data storage and performance.
• Integrate CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent tools for Databricks deployments.
• Ensure data quality, lineage, and governance using Unity Catalog or Azure Purview.
• Use Airflow, Azure Data Factory, or Databricks Workflows for scheduling and orchestration.
• Apply strong SQL, data modeling, and ETL/ELT concepts in solution design.
• Collaborate with cross-functional teams to translate business requirements into technical deliverables.
  

Required Skills & Qualifications

• 7+ years of experience in Data Engineering, with a strong focus on Databricks.
• Expertise in PySpark, Spark SQL, and optionally Scala.
• Hands-on experience with Databricks components : Workspace, Jobs, DLT, Repos, Unity Catalog.
• Strong understanding of Delta Lake and Lakehouse architectures.
• Cloud proficiency in Azure (preferred), AWS, or GCP.
• Experience in CI/CD pipeline automation for Databricks deployments.
• Familiarity with orchestration tools (ADF, Airflow, Databricks Workflows).
• Strong grasp of data governance, metadata management, and lineage.
• Excellent communication and stakeholder management skills.