Lead Data Engineer

• Design and Develop Data Pipelines:

  Hands on with development, and optimisation of scalable and reusable - data pipelines in

  Azure Microsoft Fabric Synapse Data Engineering

  , leveraging both

  batch

  and

  real-time processing techniques

  . Ensure smooth integration with

  Azure Data Factory

  for orchestration and workflow management.

• Cloud Data Architecture:

  Collaborate with the

  Data Architecture team

  to design and implement robust data architectures in the

  Azure environment

  , ensuring they align with business needs while optimising performance, scalability, and cost-efficiency.

• Pipeline Optimisation :

  Continuously monitor and optimise the performance, cost, and reliability of data pipelines, ensuring efficient processing, storage, and management of large datasets.

• Cross-functional Collaboration:

  Work closely with

  data engineering teams

  ,

  analysts

  , and

  business stakeholders

  to understand data requirements, developing solutions that enable self-service analytics and support the decision-making process.

• Documentation Knowledge Sharing:

  Contribute to internal documentation, fostering a culture of knowledge-sharing. Provide

  mentorship

  and guidance to junior engineers, helping to elevate team skills and improve overall team performance.

• Microsoft Fabric Experience:

  Apply your knowledge of

  Azure Tech Stack

  on Data Engineering (

  or your willingness to learn

  )

  Fabric-based development

  to manage end-to-end data orchestration, governance, and security across cloud and on-premises systems, ensuring seamless data movement and integration across hybrid environments.

• Data Modelling Expertise:

  Leverage your deep expertise in

  Azure

  to design and implement

  data models

  , create processing pipelines, and integrate with other

  Azure services

  like

  Data Lake

  and

  Synapse

  to support data storage and analytics needs.

Required Skills and Qualifications:

• Experience with Azure Ecosystem (Preferably Synapse):

  5+ years of hands-on experience with

  Azure Ecosystem

  , including

  Synapse

  ,

  Spark

  ,

  OneLake

  , and other Fabric tools. Expertise in optimising

  Fabric notebooks

  and efficiently managing large-scale data workloads.

• Proficiency in Azure Data Factory:

  Strong experience with designing and orchestrating complex data pipelines using

  Azure Data Factory

  , with an emphasis on seamless data flow integration across various Azure services.

• Familiarity with Microsoft Fabric:

  A working knowledge or eagerness to learn

  Azure Data Fabric

  , focusing on cross-platform data orchestration, governance, and security.

• Advanced Data Engineering Skills:

  Extensive experience in data engineering, including the design and implementation of

  ETL processes

  and working with large datasets. Proven expertise in

  data quality

  , monitoring, and testing practices.

• Cloud Architecture Design Expertise:

  Experience designing and implementing

  data architectures

  in the

  Azure ecosystem

  , including tools such as

  Data Lake

  ,

  Synapse

  , and

  Azure Storage

  .

• SQL and Data Modelling Expertise:

  Strong skills in

  SQL

  and

  data modelling

  , with the ability to design optimised data structures, tables, and views. Knowledge of both transactional and analytical data modelling.

• Collaboration and Communication Skills:

  Strong ability to work cross-functionally with teams from various domains. Ability to communicate complex technical concepts to both technical and non-technical stakeholders.

• Cost Optimisation:

  Proven experience optimising

  data engineering processes

  and

  Azure resources

  for both performance and cost, particularly in large-scale cloud environments.

Preferred Skills:

• Data Lakehouse Experience:

  Familiarity with

  Data Lakehouse

  architectures, particularly with tools like

  Delta Lake

  ,

  OneLake

  , etc.

• Azure Ecosystem Familiarity:

  Knowledge of

  Azure s full ecosystem

  for end-to-end data integration and ETL processes.

• Proficiency in PySpark and Python:

  Expertise in

  PySpark

  for data processing tasks, with a solid foundation in

  Python

  .

• Fabric Integration:

  Familiarity with

  Fabric

  and how it integrates with other services within the

  Azure ecosystem

  .

• Databricks Experience:

  Experience with

  Databricks

  is a plus.