Lead Data Engineer

6 - 11 years

7 - 11 Lacs

Posted:Just now| Platform: Naukri logo

Apply

Work Mode

Work from Office

Job Type

Full Time

Job Description

The Lead Data Engineer helps architect and lead the development of enterprise-scale data platforms and advanced analytics solutions across multiple business units and subject areas. In addition, they assist the project delivery team with scoping and planning integration projects and contribute to the design, implementation, and support of the data artifacts in the data lakehouse and databases. This role combines deep technical expertise with leadership and mentoring responsibilities, driving innovation and best practices across the data engineering function.

Job Description

About the Role:

  • Support the Solution and Data Architect(s) to help lead the design, development, and optimization of scalable data platforms and analytics solutions that support global data strategy.
  • Architect and implement robust, scalable, metadata-driven pipelines in Databricks using Delta Lake, Unity Catalog, and declarative workflows.
  • Oversee the development and maintenance of Azure SQL database objects using advanced T-SQL for data quality and insight generation.
  • Drive performance, reliability and observability of real-time data processing using Spark Structured Streaming and Change Data Feed strategies.
  • Own CI/CD processes and infrastructure automation using Databricks Asset Bundles, Azure DevOps, and Infrastructure as Code.
  • Define and enforce standards for reusable, metadata-driven integration patterns, including Unity Catalog governance, Declarative Pipeline design, and data quality across the engineering team.
  • Manage and mentor a small team of data engineers, providing technical leadership, pair programming support, conducting design/code reviews, guiding technical decisions and aligning data engineering efforts with strategic goals
  • Collaborate with cross-functional stakeholders to translate business and technical requirements into scalable data solutions, assisting the project delivery team to scope, plan and deliver new and enhanced data integrations
  • Champion innovation, best practices, and continuous improvement in data engineering, while mentoring team members and fostering technical growth.
  • Evaluate and introduce emerging technologies to enhance platform capabilities

About You:

  • 6+ years in cloud-native data engineering, including 3+ years with Databricks and Delta Lake on Azure and 2+ years in technical leadership or architecture roles.
  • Deep experience with Databricks ecosystem: Delta Lake, Unity Catalog, Declarative Pipelines, Spark Structured Streaming, Change Data Feed, Serverless SQL, and cluster optimization.
  • Expert-level SQL skills for designing performant queries, optimizing data models, and generating insights from large datasets.
  • Proficient in PySpark, Scala, R, and Python, with experience mentoring teams and promoting best practices.
  • Strong understanding of Lakehouse architecture, data integration patterns, and automation strategies.
  • Hands-on experience with CI/CD, Git, Databricks Asset Bundles, and Infrastructure as Code, driving DevOps practices in data engineering.
  • Skilled in requirements gathering and solution design, translating business needs into technical specifications.
  • Demonstrated leadership in troubleshooting complex data and code issues, performing root cause analysis, and implementing long-term fixes.
  • Exposure to machine learning pipelines, data mesh architectures, and test-driven development (pytest, nUnit, tSQLt).
  • Knowledge of real-time integrations and IoT solutions (Azure Event Hubs, Service Bus, IoT Core).
  • Familiarity with data governance frameworks (data lineage, cataloging, GDPR, HIPAA compliance).
  • Holds a degree in IT, Engineering, Computer Science, or quantitative disciplines.

Mock Interview

Practice Video Interview with JobPe AI

Start Data Engineer Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
Cushman & Wakefield logo
Cushman & Wakefield

Real Estate

Chicago IL

RecommendedJobs for You

hubli, mangaluru, mysuru, bengaluru, belgaum

chennai, tamil nadu, india

gurugram, haryana, india

new delhi, delhi, india

amritsar, punjab, india

cuttack, odisha, india

bhubaneswar, odisha, india

guwahati, assam, india