Lead Data Engineer

Key Responsibilities:

• Design & Development:

• Lead the architecture and implementation of an enterprise-grade

  data platform

  using

  Microsoft Fabric

  and related

  Azure services

  .
• Design and implement

  Medallion architecture

  (Bronze/Silver/Gold layers) to manage raw, validated, and curated data.
• Develop

  real-time data ingestion

  , validation, and transformation pipelines supporting heterogeneous file formats.
• Build and orchestrate

  event-driven workflows

  using

  serverless Azure Functions

  ,

  Event Hub

  , and

  Durable Functions

  .
• Integrate

  Fabric Lakehouses

  and

  Synapse Data Warehousing

  with transactional systems and UI components.
• Implement

  schema evolution

  ,

  metadata management

  , and

  lineage tracking

  across the platform.
• Enable business self-service through configurable

  rules engines

  and

  validation UIs

  .
• Ensure

  role-based access control (RBAC)

  ,

  end-to-end encryption

  , and

  audit trails

  across all layers.
• Build

  CI/CD pipelines

  and

  monitoring

  for automated deployments and observability.

• Collaboration & Leadership:

• Collaborate with

  UI engineers

  ,

  solution architects

  ,

  product owners

  , and

  compliance stakeholders

  .
• Lead and mentor a team of

  data engineers

  ; establish

  coding

  and

  documentation standards

  .

Required Skills and Qualifications:

• Experience:

• 8-12 years of hands-on experience in

  data engineering

  , including

  3+ years

  in a leadership or architecture role.

• Core Skills:

• Strong expertise in

  Microsoft Azure

  , especially:

• Microsoft Fabric

  (Data Lakehouses, Pipelines, Real-Time Analytics, Synapse).

• Azure Data Factory

  ,

  Azure Functions

  ,

  Event Hub / Service Bus

  .

• Delta Lake

  on

  ADLS Gen2

  .

• Azure API Management

  and

  App Services

  .
• Advanced knowledge of

  Python

  ,

  PySpark

  ,

  SQL

  , and

  workflow orchestration

  .
• Experience building

  event-driven

  ,

  serverless architectures

  in a production environment.
• Deep understanding of

  schema-agnostic ingestion

  , validation logic, and dynamic rule engines.
• Knowledge of

  data privacy

  ,

  regulatory compliance

  ,

  lineage

  , and

  auditability

  .
• Experience integrating with

  transactional UIs

  and

  REST APIs

  for CRUD operations.
• Proven track record delivering large-scale,

  cloud-native data platforms

  .

Good to Have:

• Domain knowledge in

  insurance

  ,

  financial services

  , or

  regulatory systems

  .
• Familiarity with

  RE7/FSRI

  ,

  master data integration

  , or third-party systems.
• Experience with

  retention

  ,

  premium

  , or

  commission calculations

  .
• Prior exposure to

  Microsoft Purview

  ,

  Azure Monitor

  , and

  Azure Key Vault

  .
• Understanding of

  multilingual support

  ,

  global data residency compliance

  , and

  deployment automation

  .