Senior GCP Data Engineer

GCP Data Engineer,

Databricks

Key Responsibilities:

• Designing and developing data pipelines:

  Design and implement scalable and efficient data pipelines using 

  GCP-native services

   (e.g., Cloud Composer, Dataflow, BigQuery) and tools like 

  Databricks

  , 

  PySpark

  , and 

  Scala

  . This includes data ingestion, transformation, and loading (ETL/ELT) processes.

• Data modeling and database design:

  Develop data models and schema designs to support efficient data storage and analytics using tools like 

  BigQuery

  , 

  Cloud Storage

  , or other GCP-compatible storage solutions.

• Data integration and orchestration:

  Orchestrate and schedule complex data workflows using 

  Cloud Composer (Apache Airflow)

   or similar orchestration tools. Manage end-to-end data integration across cloud and on-premises systems.

• Data quality and governance:

  Implement data quality checks, validation rules, and governance processes to ensure data accuracy, integrity, and compliance with organizational standards and external regulations.

• Performance optimization:

  Optimize pipelines and queries to enhance performance and reduce processing time, including tuning Spark jobs, SQL queries, and leveraging caching mechanisms or parallel processing in GCP.

• Monitoring and troubleshooting:

  Monitor data pipeline performance using 

  GCP operations suite (formerly Stackdriver)

   or other monitoring tools. Identify bottlenecks and troubleshoot ingestion, transformation, or loading issues.

• Documentation and collaboration:

  Maintain clear and comprehensive documentation for data flows, ETL logic, and pipeline configurations. Collaborate closely with data scientists, business analysts, and product owners to understand requirements and deliver data engineering solutions.

Skills and Qualifications:

• 5+ years of experience

   in a Data Engineer role with exposure to large-scale data processing.
• Strong hands-on experience with 

  Google Cloud Platform (GCP)

  , particularly services like 

  BigQuery

  , 

  Cloud Storage

  , 

  Dataflow

  , and 

  Cloud Composer

  .
• Proficient in 

  Python

   and/or 

  Scala

  , with a strong grasp of 

  PySpark

  .
• Experience working with 

  Databricks

   in a cloud environment.
• Solid experience building and maintaining big data pipelines, architectures, and data sets.
• Strong knowledge of 

  Informatica

   for ETL/ELT processes.
• Proven track record of manipulating, processing, and extracting value from large-scale, unstructured datasets.
• Working knowledge of stream processing and scalable data stores (e.g., Kafka, Pub/Sub, BigQuery).
• Solid understanding of 

  data modeling

   concepts and best practices in both OLTP and OLAP systems.
• Familiarity with 

  data quality frameworks

  , governance policies, and compliance standards.
• Skilled in 

  performance tuning

  , job optimization, and cost-efficient cloud architecture design.
• Excellent communication and collaboration skills to work effectively in cross-functional and client-facing roles.
• Bachelor's degree in 

  Computer Science

  , 

  Information Systems

  , or a related field (Mathematics, Engineering, etc.).
• Bonus: Experience with 

  distributed computing frameworks

   like Hadoop and Spark.