Lead Data Engineer

Overview

We are seeking a highly skilled Lead Data Engineer to architect, build, and scale enterprise-grade data platforms and pipelines. The ideal candidate has deep expertise in modern data engineering, cloud-native services, real-time data ingestion & CDC frameworks. This role involves leading complex data initiatives, guiding engineering teams, and collaborating with cross-functional stakeholders to deliver robust, scalable, and secure data solutions.

Key Responsibilities

Data Architecture & Pipeline Development

• 
  Design and build scalable ETL/ELT pipelines using Python / PySpark.

• 
  Architect and implement batch and real-time data pipelines using Kafka, Kafka Connect, and Debezium.

• 
  Develop and optimize data ingestion workflows across multiple databases (MySQL, Postgres, MongoDB).

• 
  Build and manage Apache Airflow DAGs for end-to-end pipeline orchestration.

Cloud Data Engineering

• 
  Lead data platform development on GCP (BigQuery, Data Proc, Data Stream ).

• 
  Implement data ingestion, orchestration, and transformation at scale using Pub/Sub, Cloud Functions/Cloud Run, Dataflow, and GKE.

• 
  Own data warehouse design and optimization across Snowflake and BigQuery.

Requirements

Required Skills & Experience

Core Technical Skills

• 
  Strong proficiency in Python, SQL, PySpark.

• 
  Hands-on expertise with Kafka, Kafka Connect, Debezium, Airflow, Databricks.

• 
  Deep experience with BigQuery, Snowflake, MySQL, Postgres, MongoDB.

• 
  Solid understanding of vector data stores and search indexing.

• 
  Knowledge of GCP services like Big Query, Cloud Functions, Cloud Run, Data Flow, Data Proc, Data Stream, etc..

Good to have

• 
  Certifications:

• 
  GCP Professional Data Engineer

• 
  Elastic Certified Engineer

• 
  AI

• 
  Gemini Enterprise

• 
  Vertex AI Agent Builder

• 
  ADK

Non-Technical & Leadership Skills

• 
  Communication: Exceptional verbal and written communication skills, with the ability to articulate complex technical concepts to both technical and non-technical audiences.

• 
  Mentorship & Coaching: Proven experience in mentoring junior and mid-level engineers, fostering a culture of continuous learning and growth.

• 
  Problem-Solving: Strong analytical and debugging skills, with a proactive approach to identifying and resolving technical roadblocks.

• 
  Ownership & Accountability: Demonstrates a high level of responsibility for project outcomes, system reliability, and code quality.

• 
  Agile Proficiency: Deep understanding and practical experience with Agile methodologies (Scrum/Kanban).

• 
  Stakeholder Management: Ability to effectively manage expectations and build consensus across different teams.

Qualifications

• 
  Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).