Lead Data Engineer

Job Overview

We are seeking a highly skilled Data Engineering Lead to design, build, and optimize scalable data pipelines for our Data Platform. The ideal candidate has strong hands-on experience in PySpark, distributed data processing, and cloud-based data solutions, and expertise in building Data Pipelines.

This role requires leadership skills, technical depth, and the ability to collaborate with cross-functional teams to drive strategic data initiatives.

Key Responsibilities

• Lead the design, development, and optimization of data pipelines using low Code/No code tool
• Oversee end-to-end data workflowsfrom ingestion, transformation, and storage to data lakein both batch and streaming environments.
• Work closely with the product and other teams to understand data requirements and translate them into robust engineering solutions.
• Ensure data quality, governance, security, and compliance across all data pipelines.
• Mentor and guide junior data engineers, performing code reviews and establishing engineering best practices.
• Drive improvements in performance, reliability, and scalability of data infrastructure.
• Troubleshoot complex data issues and implement long-term fixes.

Required Skills & Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or related field.
• At least 4 years of hands-on experience in data engineering
• Strong proficiency in distributed data processing frameworks (Spark, PySpark).
• Strong SQL skills and experience working with relational and NoSQL databases.
• Demonstrates advanced proficiency in Python for building scalable data pipelines, automating workflows
• Experience with cloud platforms such as AWS, Azure, GCP
• Experience building scalable ETL/ELT data pipelines and data lakes/warehouses.
• Familiarity with workflow orchestration tools (Airflow, ADF, Glue Workflow, etc.)
• Solid understanding of CI/CD, version control, and DevOps practices.
• Experience working in agile, collaborative environments.
• Excellent problem-solving, communication, and leadership skills.