N

Lead Data Engineer

Narpavi Soft Solutions

15 - 21 Lacs

Bengaluru

Posted:5 days ago

Job Description

Role Summary

Lead a Python-first data engineering function, owning production-grade codebases and frameworks for a Databricks Lakehouse platform, while setting direction for pipeline architecture, data modelling and performance.

Key Responsibilities

  • Build production Python ETL/ELT pipelines, reusable packages, schema validation and testing frameworks (pytest, CI/CD).
  • Write performant Python using OOP, generators, decorators and concurrency (multiprocessing/asyncio); profile, debug and optimize pipeline code.
  • Set Python engineering standards for reviews, packaging/versioning, logging and exception handling.
  • Design Bronze/Silver/Gold Medallion pipelines on Databricks using Delta Lake, Structured Streaming, Auto Loader and Unity Catalog.
  • Own dimensional modelling (star/snowflake, SCD); write/review advanced T-SQL, MERGE/upsert logic and execution-plan-based tuning.
  • Partner on ETL/ELT, orchestration and cost/performance decisions; mentor Data Engineers and Senior Data Engineers.

Must-Have Qualifications

  • 8–12 years in data engineering, including 2+ years in technical leadership or mentoring.
  • Strong Python: OOP, generators, decorators, concurrency, exception handling, pytest and production pipeline frameworks.
  • Hands-on Databricks: Delta Lake (ACID, time travel, OPTIMIZE/ZORDER/VACUUM), Structured Streaming, Auto Loader and Unity Catalog.
  • Spark performance tuning, data skew, cluster sizing/autoscaling and cost optimization.
  • Dimensional modelling: star/snowflake, SCD Types 1/2/3, fact/dimension grain and conformed dimensions; working knowledge of Data Vault and

Kimball/Inmon.

  • Advanced T-SQL, indexing and query tuning; Medallion architecture and ETL/ELT and batch/streaming trade-offs.
  • Orchestration using Databricks Workflows, Airflow or ADF on Azure or AWS.

Good to Have & Education

  • PCEP/PCAP or equivalent Python certification; Databricks Certified Data Engineer Professional.
  • Data governance/compliance (GDPR/HIPAA), FinOps and experience leading/mentoring 3+ engineers.
  • Bachelor’s/Master’s in Computer Science, Engineering or related field, or equivalent practical experience.
  • Strong stakeholder communication and ability to explain technical trade-offs and own enterprise-scale design decisions

Pay: ₹1,500,000.00 - ₹2,100,000.00 per year

Work Location: In person

Skills Required

Python Data Engineering Data Modelling ETL Testing Frameworks Ci/cd Packaging T-sql Technical Leadership Spark Performance Tuning Cost Optimization

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
N
Narpavi Soft Solutions

RecommendedJobs for You

Accord Infosystems logo

Lead Data Engineer

Accord Infosystems

Mumbai

Ecolab logo

Bengaluru

Aptean logo

Bengaluru

Hiringhood logo

Hyderabad

Echidna Software logo

Lead Data Engineer

Echidna Software

Bengaluru

Delta System & Software, Inc. logo

Lead Data Engineer

Delta System & Software, Inc.

Chennai

T

India

Neurealm logo

India