Senior Data Engineer

Role overview

Microsoft Fabric

What you’ll do

• Design and implement scalable data pipelines (batch + streaming) from diverse sources (REST, SFTP, RDBMS, Kafka/Event Hubs/Service Bus) into a lakehouse and

  OneLake.

• Model and curate datasets using medallion architecture; build reusable frameworks for ingestion, schema evolution, and incremental processing.
• Write efficient transformations in Spark (PySpark/SQL) and/or KQL; create materialized views, update policies, and optimization strategies for cost/perf.
• Implement CDC, watermarking, late-arrivals handling, and idempotent writes for append/merge scenarios.
• Enforce data quality, observability, and lineage (DQ rules, expectations, SLAs, alerts, metadata catalogs).
• Apply security & governance best practices (PII hashing/tokenization, access controls, secrets management).
• Productionize workloads with orchestration (Airflow/ADF/Azure Synapse/Step Functions/Glue), CI/CD, testing, and rollout strategies.
• Partner with product/analytics teams to define SLAs, table contracts, and consumption patterns; create reliable semantic layers.
• Troubleshoot performance, skew, and reliability issues; tune storage (Delta/Parquet/Iceberg) and compute configurations.

What you’ll bring

• 6+ years of data engineering experience (title flexible: Data Engineer / Senior Data Engineer).
• Strong SQL and one of Python/Scala. Deep familiarity with Spark (PySpark/SQL) and distributed data patterns.
• Hands-on with one or more clouds (Azure/AWS/GCP) and a lakehouse stack (e.g., Databricks, Delta Lake, Fabric Lakehouse/Eventhouse, Synapse, BigQuery/Snowflake a plus).
• Streaming experience: Kafka/Confluent, Azure Event Hubs, or Service Bus; schema registry, exactly-once/at-least-once semantics.
• Solid understanding of medallion architecture, CDC, SCD, upserts/merge, partitioning, Z-ordering, compaction, and vacuum.
• Orchestration & DevOps: Airflow/ADF/Glue/Step Functions; Git-based workflows, unit/integration tests, environments, and IaC (Terraform/ARM/CDK) preferred.
• Data quality & governance: expectations/testing, lineage/metadata, RBAC/ABAC, PII protection (hashing/salting/tokenization).
• Comfortable owning services in production: monitoring, alerting, SLIs/SLOs, on-call rotation.

What You’ve Done (Must-Haves)

• 5+ years in data engineering with

  cloud data platforms

  (Azure preferred).
• Hands-on with

  Microsoft Fabric

  components:

  Eventhouse (KQL)

  ,

  Lakehouse (Delta on OneLake)

  ,

  Spark notebooks

  ,

  Data Factory (Fabric pipelines)

  ,

  Power BI

  (including Direct Lake).
• Solid

  SQL/KQL/PySpark

  ; comfort with nested JSON, mv-expand, update policies, materialized views, partitioning.
• Built production-grade streaming + batch pipelines; handled late/duplicate events, watermarking, and idempotency.
• Strong grasp of

  data modeling

  , performance tuning, and

  data quality

  (unit tests, anomaly checks, SLAs).

Nice to Have

• Confluent Kafka private networking patterns; CDC from operational stores.
• Azure ecosystem:

  ADLS/OneLake

  , Key Vault, AAD, Purview, Event Hubs, Service Bus.
• MLOps/feature store basics; Python packaging & testing (pytest).
• Governance & compliance (GDPR/CCPA), PII handling, and secrets management.

Tech Stack You’ll Touch

• Microsoft Fabric:

  Eventhouse/KQL, Lakehouse/Delta, Spark notebooks, Data Factory, Power BI (Direct Lake)

• Azure:

  Event Hubs, Service Bus, AAD, Key Vault, Purview

• Langs/Tools:

  SQL,

  KQL

  ,

  PySpark

  , Python, Git, CI/CD (ADO/GitHub)