Job
Description
Job Title: Lead Data Engineer Location: Pune (On-Site) Experience Level: 8+ Years Employment Type: Full-time Job Summary: We are seeking a highly skilled Lead Data Engineer with a strong background in Python, Apache Spark, and Apache Airflow to join our growing data team. You will lead the design, development, and deployment of scalable data pipelines and systems, ensuring high data quality and reliability for advanced analytics, AI/ML, and business intelligence solutions. Key Responsibilities: Lead the end-to-end development of scalable, high-performance data pipelines using Python , Apache Spark , and Apache Airflow . Collaborate with data scientists, analysts, and other engineering teams to define data architecture and infrastructure strategies. Develop and maintain ETL/ELT workflows to ingest data from diverse structured and unstructured sources. Ensure data quality , governance , lineage , and observability across the data ecosystem. Optimize Spark jobs for performance and cost-efficiency on distributed systems. Design and implement data models , data lakes , and data warehouses (preferably on AWS, GCP, or Azure). Mentor and guide junior engineers on coding best practices, architectural decisions, and performance tuning. Monitor production workflows, troubleshoot failures, and implement preventive measures for data reliability. Drive the adoption of best practices in data engineering , code review , CI/CD , and infrastructure as code . Required Qualifications: 8+ years of experience in software/data engineering with a strong focus on data pipeline development . Expertise in Python programming for data processing, scripting, and orchestration. Hands-on experience with Apache Spark (PySpark, SparkSQL) for distributed data processing. Strong knowledge of Apache Airflow for workflow orchestration and scheduling. Proficiency with SQL and working with relational and NoSQL databases. Experience working with cloud platforms (AWS, Azure, or GCP) and tools like S3, Redshift, BigQuery, Databricks, etc. Experience with CI/CD pipelines , Docker , and version control (e.g., Git). Strong analytical and problem-solving skills with attention to detail. Preferred Qualifications: Experience with data lakehouse architecture and tools like Delta Lake , Iceberg , or Hudi . Familiarity with Kafka , Snowflake , or dbt is a plus. Exposure to data governance tools and frameworks. Understanding of ML pipelines , feature stores , or MLOps is a bonus. Show more Show less
