Lead Data Engineer

Location-Remote

Notice Period-Immediate-10 days

The Lead Data Engineer will architect, build, and operate scalable batch and real-time data pipelines on Google Cloud Platform. The role owns end-to-end data integration using Python, Apache Spark, Kafka, and GCP services like Dataflow, BigQuery, Cloud Composer, Cloud Functions, and Cloud Run. The engineer will ensure data quality, security, redaction/DLP, and robust CI/CD practices while optimizing performance and cost. They will collaborate with data scientists and analysts, design APIs with FastAPI/APIGEE, and guide deployments on GKE/Cloud Run. The position requires strong SQL skills, cloud migration experience, and excellent documentation and communication.

Required Qualifications

• Expert-level Python for backend/data engineering
• Hands-on Google Cloud Platform (Dataflow, BigQuery, Cloud Composer, Cloud Functions, Cloud Run, GCS, IAM)
• Design and development of ETL/ELT pipelines (batch and streaming)
• Apache Spark for large-scale processing
• Apache Kafka for messaging/streaming
• Orchestration with Airflow/Cloud Composer (DAG design, scheduling, monitoring)
• Strong SQL with enterprise RDBMS (SQL Server, Oracle, PostgreSQL)
• Git/GitHub and CI/CD for data projects
• Deployments on GKE and Cloud Run; understanding of autoscaling and load balancers
• API development with Python FastAPI; APIGEE proxy management
• Data quality frameworks, validation rules, monitoring
• Security: IAM roles/policies, data redaction, DLP
• Data ingestion from varied sources; transformation and cleansing
• Documentation of pipelines, data flows, and operations
• Experience migrating on-prem data to cloud

Responsibilities

• Design, develop, test, and maintain scalable ETL/ELT data pipelines in Python
• Architect data solutions using Kafka, GCP services (Dataflow, BigQuery, Composer, Functions, Cloud Run/GKE, GCS, IAM), DBT, and related tools
• Implement streaming and batch processing; ensure autoscaling, reliability, and cost efficiency
• Develop and manage Airflow/Cloud Composer DAGs for orchestration
• Build APIs with FastAPI and manage APIs via APIGEE; integrate load balancers when required
• Ingest data from diverse sources; apply transformation, cleansing, and enrichment
• Implement data quality checks, validation rules, observability, and monitoring
• Apply security best practices including IAM, data redaction, and DLP
• Write and optimize complex SQL for extraction, validation, and analytics
• Manage code in GitHub; implement CI/CD pipelines for data workloads
• Deploy and operate workloads on GKE and Cloud Run
• Collaborate with data science/analytics teams to deliver fit-for-purpose data products
• Support documentation of pipeline designs, data flows, and operational runbooks
• Lead or contribute to on-prem to cloud data migration initiatives