GCP Data Engineer - (Databricks)

Role Overview

We are looking for a Senior Data Engineer with strong hands-on experience in building and optimizing scalable data pipelines on Google Cloud Platform (GCP) and Databricks. This role requires deep technical expertise in data engineering, pipeline development, and cloud-native solutions, with the ability to work closely with cross-functional teams to deliver high-quality data systems.

Key Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines on GCP and Databricks.
• Develop and maintain ETL/ELT workflows for diverse data sources.
• Build robust data ingestion frameworks and support data migration initiatives.
• Develop and optimize Spark/PySpark workloads in Databricks, including notebooks, jobs, and Delta Lake tables.
• Manage workflow orchestration, scheduling, and pipeline dependencies (Cloud Composer, Databricks Workflows).
• Ensure efficient and reliable data processing across environments.
• Optimize query performance and data processing jobs, including Databricks cluster and job tuning.
• Implement data validation, quality, and reconciliation frameworks.
• Monitor pipelines to ensure availability, reliability, and performance.
• Collaborate with cross-functional teams to translate business requirements into solutions.
• Contribute to data architecture and design discussions.
• Follow best practices for version control, CI/CD, and deployments.
• Ensure compliance with security, governance, and data standards, including access control and cataloging in Databricks.
• Support production systems through troubleshooting and continuous improvement.

Must-Have Skills

• 6-8 years of experience in data engineering.
• Strong hands-on experience with GCP (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Hands-on experience with Databricks, including Delta Lake, Databricks Workflows/Jobs, and notebook-based development.
• Experience with Spark /PySpark and SQL optimization.
• Hands-on with Cloud Composer (Airflow).
• Strong understanding of data modeling and pipeline design.
• Experience with batch + streaming data processing.
• Familiarity with CI/CD and Git-based workflows.
• Strong problem-solving and analytical skills.