Technology & Transformation Engineering - Consultant/Senior Consultant

Key Responsibilities

• Design, develop, and maintain

  ETL/ELT pipelines

  using

  Azure Data Factory

  and

  Azure Databricks

  .
• Work with structured and unstructured data from diverse sources such as SQL databases, APIs, SAP, flat files, and cloud storage.
• Implement

  Delta Lake

  , data ingestion frameworks, and

  real-time/streaming pipelines

  where required.
• Optimize Databricks notebooks and pipelines for performance, scalability, and cost efficiency.
• Ensure high data quality through validation, reconciliation, and automated checks.
• Collaborate with product owners, architects, and business teams to understand data requirements.
• Implement best practices in

  CI/CD using Azure DevOps

  , version control, and environment management.
• Support production pipelines, troubleshoot issues, and ensure timely resolution.
• Contribute to documentation, design specs, and operational playbooks.

Required Skills

• Strong hands-on experience with

  Azure Databricks

  and

  PySpark

  .
• Expertise in

  Azure Data Factory

  (ADF) pipeline development.
• Practical knowledge of

  Delta Lake

  , schema evolution, and data lake optimization.
• Experience with

  SQL

  , performance tuning, and data modeling.
• Working knowledge of Azure services such as

  ADLS, Azure Key Vault, Azure Monitor

  , etc.
• Familiarity with

  Azure DevOps

  (Repos, Pipelines, CI/CD).
• Strong debugging, problem-solving, and analytical skills.

Good to Have

• Experience with

  Lakehouse architecture

  .
• Exposure to

  Synapse

  ,

  Unity Catalog

  ,

  Databricks Jobs

  , and

  Cluster management

  .
• Knowledge of

  real-time streaming

  (Event Hub, Kafka, Auto Loader).
• Understanding of

  data governance

  ,

  security

  , and

  access management

  .