Lead Data Engineer

Job Summary:

Lead Data Engineer

Key Responsibilities:

• Oversee the

  entire data infrastructure

  to ensure scalability, operational efficiency, and resiliency.

• Mentor junior data engineers

  within the organization, fostering their growth and skill development.
• Design, develop, and maintain robust

  data pipelines and ETL processes

  using a comprehensive suite of

  Microsoft Azure services

  , including but not limited to Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Fabric.
• Effectively utilize

  Azure data storage accounts

  (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage) for organizing and maintaining data pipeline outputs.
• Collaborate extensively with data scientists, data analysts, data architects, and other stakeholders to deeply understand data requirements and deliver high-quality, impactful data solutions.
• Optimize data pipelines within the Azure environment for maximum performance, scalability, and reliability.
• Ensure stringent

  data quality and integrity

  through the implementation of advanced data validation techniques and frameworks.
• Develop and maintain thorough documentation for all data processes, configurations, and best practices.
• Proactively monitor and troubleshoot data pipeline issues to ensure timely resolution and minimize downtime.
• Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge and competitive.
• Manage the

  CI/CD (Continuous Integration/Continuous Deployment) process

  for deploying and maintaining data solutions.

Background & Skills:

• Proven experience in a senior or lead role, utilizing software engineering principles for data infrastructure.
• Strong expertise in designing, developing, and maintaining fully automated data transformation pipelines.
• In-depth practical experience with a wide variety of storage and computation technologies.
• Extensive hands-on experience with

  Microsoft Azure data services

  such as Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Fabric.
• Proficient in using

  Azure data storage solutions

  like Azure Data Lake Storage Gen 2 and Azure Blob storage.
• Demonstrated ability to optimize data pipelines for performance, scalability, and reliability in a cloud environment.
• Experience with data quality and integrity assurance through validation techniques.
• Familiarity with CI/CD processes for data solutions.
• Excellent collaboration and communication skills for working with cross-functional teams (data scientists, analysts, architects).
• Strong problem-solving abilities and a commitment to continuous learning in emerging data technologies.