Data Engineering

Microsoft Fabric

Key Responsibilities

• Build data pipelines

  using

  PySpark

  on

  Databricks

  /

  Fabric Notebooks

  , ingesting data from APIs, files, and DBs into the

  Fabric Lakehouse

  /

  Azure Data Lake

  .

• Develop ELT/ETL workflows

  : transform and curate bronze silver gold layers; implement partitioning, Delta tables, and performance tuning.

• Data modeling & warehousing

  : design dimensional models (star/snowflake), views, and semantic layers for BI & analytics in

  Synapse

  /

  Fabric

  .

• SQL development

  : write complex queries, stored procedures, and optimize execution plans; enforce coding standards.

• Orchestration & CI/CD

  : schedule jobs with

  Fabric pipelines

  /

  Azure Data Factory

  ; implement CI/CD for notebooks, SQL, and infrastructure (Git, YAML pipelines).

• Data quality & governance

  : implement validation checks, lineage, cataloging, and security (RBAC, rowlevel/columnlevel security).

• Performance & cost optimization

  : tune clusters, caching, autoscaling, job concurrency; monitor and manage workspace costs.

• Collaboration

  : partner with analysts, data scientists, and product teams to deliver reliable datasets and SLAs.

MustHave Skills

• Microsoft Fabric

  (Lakehouse, Data Engineering, Pipelines) or strong

  Azure Synapse

  experience with readiness to work in Fabric.

• Databricks

  (Delta Lake, notebooks, cluster management).

• PySpark

  (RDD/DataFrame APIs, UDFs, joins, windowing, checkpointing).

• SQL

  (TSQL/ANSI SQL, query optimization, indexing).

• Azure

  data services:

  Azure Data Lake Storage

  ,

  Key Vault

  ,

  Azure DevOps

  (repos, pipelines).

• Data modeling

  : dimensional modeling, data vault basics, semantic modeling for BI.

NicetoHave

• Orchestration

  : Azure Data Factory, Airflow.

• BI

  : Power BI (Direct Lake/Import).

• Streaming

  : Structured Streaming, Event Hub, Kafka.

• Automation/IaC

  : Terraform/Bicep for Azure resources.

• Testing

  : Great Expectations, dbt tests, unit tests for PySpark.

• Security & Governance

  : Purview (catalog, lineage, classification).