Azure Data Lead

Job ID:

Location:

The Role:

Key Responsibilities

• Technical Leadership:
• Lead the design, development, and implementation of scalable and reliable data pipelines using Databricks, Spark, and other relevant technologies.
• Define and enforce data engineering best practices, coding standards, and architectural patterns.
• Provide technical guidance and mentorship to junior and mid-level data engineers.
• Conduct code reviews and ensure the quality, performance, and maintainability of data solutions.
• Databricks Expertise:
• Architect and implement data solutions on the Databricks platform, including Databricks Lakehouse, Delta Lake, and Unity Catalog.
• Optimize Spark workloads for performance and cost efficiency on Databricks.
• Develop and manage Databricks notebooks, jobs, and workflows.
• Proficiently use Databricks features such as Delta Live Tables (DLT), Photon, and SQL Analytics.
• Pipeline Development & Operations:
• Develop, test, and deploy robust ETL/ELT pipelines for data ingestion, transformation, and loading from various sources (e.g., relational databases, APIs, streaming data).
• Implement monitoring, alerting, and logging for data pipelines to ensure operational excellence.
• Troubleshoot and resolve complex data-related issues.
• Collaboration & Communication:
• Work closely with cross-functional teams including product managers, data scientists, and software engineers.
• Communicate complex technical concepts clearly to both technical and non-technical stakeholders.
• Stay updated with industry trends and emerging technologies in data engineering and Databricks
  

Primary Skills

• Extensive hands-on experience with Databricks platform, including Databricks Workspace, Spark on Databricks, Delta Lake, and Unity Catalog.
• Strong proficiency in optimizing Spark jobs and understanding Spark architecture.
• Experience with Databricks features like Delta Live Tables (DLT), Photon, and Databricks SQL Analytics.
• Deep understanding of data warehousing concepts, dimensional modeling, and data lake architectures.