Data Engineer

Role Overview

Azure Databricks

Key Responsibilities

• Design, build, and maintain scalable data pipelines using

  Azure Data Factory (ADF)

  and

  Microsoft Fabric Pipelines / Dataflows

  .
• Develop data transformations using

  PySpark / Spark SQL

  in

  Azure Databricks

  or Microsoft Fabric Notebooks.
• Build and maintain

  Lakehouse architecture

  on Microsoft Fabric (OneLake), following medallion (bronze / silver / gold) patterns or as required.
• Model data (dimensional modeling, star schema, etc.) for analytical use cases, data warehousing, and BI.
  
• Optimize performance of Spark jobs, data pipelines, and storage (partitioning, caching, indexing).
  
• Monitor, troubleshoot, and tune data pipelines for reliability, latency, and cost.
  
• Implement and enforce data quality practices, data validation, and data lineage.
  
• Collaborate with data analysts, data scientists, and business stakeholders to understand and translate data requirements.
  
• Ensure data governance, security, and access control in Microsoft Fabric, including row-level / column-level security where relevant.
  
• Automate deployments and orchestrate CI/CD for data pipelines and notebook code, using tools like Azure DevOps / Git.
  
• Document data architecture, pipeline design, data flows, and best practices.
  
• Stay up-to-date with new features in Azure Databricks, ADF, and Microsoft Fabric, and drive adoption of best practices.
  

Required Skills & Experience

• Experience

  : 38+ years (depending on seniority) in data engineering or a related role.

• Cloud

  : Strong experience with

  Azure

  , specifically Azure Data Factory, Databricks, and other Azure data services.

• Big Data / Spark

  : Proficient in

  PySpark

  (or Spark), Spark SQL, familiarity with Delta Lake or similar.

• Microsoft Fabric

  : Experience working with Fabric’s Lakehouse, Dataflows, Pipelines, Notebooks, or other Fabric components.
• Note: Microsoft Fabric is a unified data platform integrating data movement, OneLake, Spark, etc. prepzeee.wixsite.com+2unitedtechno.com+2
  

• Programming

  : Strong skills in Python, especially for data transformation and scripting.

• SQL

  : Deep understanding of SQL, relational data modeling, data warehousing principles.

• Data Modeling

  : Experience in dimensional modeling, star / snowflake schemas, or medallion architecture.

• DevOps / Automation

  : Experience with version control (Git), CI/CD pipelines for data workloads, and scheduling.

• Data Governance

  : Understanding of data governance, lineage, security, and compliance within data platforms.

• Collaboration

  : Good communication skills to work with cross-functional teams (BI, Data Science, Business).

• Problem-Solving

  : Ability to debug complex data issues, optimize performance, and improve pipeline resiliency.