Data Engineer (Azure ML ,Databricks, Data lakes)

About the Role

Data Engineers

Key Responsibilities

• Design, develop, and optimize

  ML/data transformation pipelines

  using

  Python, PySpark, and Apache Spark

  .
• Build and manage scalable

  data architectures

  for handling large and complex datasets.
• Work extensively on

  Azure Databricks

  and

  Delta Lake

  , ensuring efficient data storage and processing.
• Collaborate with cross-functional teams to deliver

  data solutions

  across multiple domains.
• 
  
• Implement best practices in

  data modeling

  ,

  data warehousing

  , and

  data governance

  for both on-prem and cloud environments.
• Continuously evaluate emerging technologies and tools to enhance data engineering capabilities.
• Troubleshoot, optimize, and ensure performance of data pipelines and analytical systems.
• Take ownership of technical deliverables, contributing to

  project leadership

  and

  independent execution

  .

Required Skills & Experience

• Mandatory:

  Strong hands-on experience with

  Databricks

  and

  Delta Lake

  .
• Proficiency in

  Python

  ,

  PySpark

  , and

  Apache Spark

  .
• Experience in

  ML pipelines

  ,

  data integration

  , and

  data pipeline optimization

  .
• Solid understanding of

  data modeling

  ,

  data warehousing

  , and

  distributed systems

  .
• Ability to manage and transform

  large-scale, multi-dimensional datasets

  .
• Strong

  problem-solving

  ,

  analytical

  , and

  process optimization

  skills.
• Up-to-date with

  modern data engineering tools

  and

  cloud data ecosystems

  .