Azure Data Engg

Data Engineer

• Design, build, and maintain scalable and efficient

  data pipelines

  using

  Databricks Notebooks

  ,

  Workflows

  , and

  Delta Live Tables (DLT)

  .
• Leverage

  Databricks SQL

  ,

  PySpark

  , and

  Python

  for data transformation, enrichment, and analytics.
• Implement and manage

  Delta Lake

  architectures to support real-time and batch data processing.
• Use

  Unity Catalog

  for data governance, access control, and metadata management.
• Orchestrate complex workflows using

  Databricks Workflows

  and integrate with external schedulers if needed.
• Optimize performance of Spark jobs and manage cluster configurations for cost and efficiency.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-grade data solutions.
• Ensure data quality, lineage, and observability using

  Databricks tools

  and best practices.
• Integrate with

  AWS S3

  or

  Google Cloud Storage (GCS)

  for data ingestion and storage.
• Follow Agile methodologies and use

  Git

  for version control and CI/CD pipelines.

Qualifications

• Bachelors degree in Computer Science, Information Technology, or a related field.
• 6+ years of experience as a Data Engineer, with

  3+ years of hands-on experience with Databricks

  .
• Strong expertise in

  Databricks Lakehouse Platform

  , including:

  • Delta Lake

  • Databricks SQL

  • Unity Catalog

  • Databricks Workflows

  • Delta Live Tables (DLT)

• Proficiency in

  PySpark

  ,

  Python

  , and

  SQL

  .
• Experience with

  cloud storage

  services like

  AWS S3

  or

  Google Cloud Storage

  .
• Solid understanding of

  data modeling

  ,

  data warehousing

  , and

  ETL/ELT

  best practices.
• Exposure to

  BI tools

  like

  Tableau

  ,

  Power BI

  , or

  Databricks SQL Dashboards

  .
• Experience with

  CI/CD pipelines

  ,

  Git

  , and

  automated testing

  for data pipelines.