Big Data Developer Senior

Key Responsibilities

• Design and develop large-scale

  data pipelines

  using GCP services (BigQuery, Dataflow, Dataproc, Pub/Sub).
• Implement batch and real-time

  ETL/ELT pipelines

  using Apache Beam and Spark.
• Manage and optimize

  BigQuery

  queries, partitioning, clustering, and cost control.
• Build distributed processing jobs on

  Dataproc (Hadoop/Spark)

  clusters.
• Develop and maintain

  streaming data pipelines

  with Pub/Sub and Dataflow.
• Work with

  Cloud Spanner

  to support highly available and globally scalable databases.
• Integrate data from various sources, manage schema evolution, and ensure data quality.
  
• Collaborate with data analysts, data scientists, and business teams to deliver scalable data solutions.
  
• Follow

  CI/CD

  , DevOps, and infrastructure-as-code best practices using tools like

  Terraform

  or

  Cloud Build

  .
• Monitor, debug, and tune data pipelines for performance and reliability.
  

Must-Have Skills

• GCP expertise

  BigQuery, Dataflow, Dataproc, Cloud Spanner, Pub/Sub.
• Strong SQL skills and performance optimization in BigQuery.
  
• Solid experience in

  streaming (real-time)

  and

  batch processing

  .
• Proficiency in

  Apache Beam

  ,

  Apache Spark

  , or similar frameworks.
• Python or Java for data processing logic.
  
• Understanding of

  data architecture

  , pipeline design patterns, and

  distributed systems

  .
• Experience with

  IAM roles

  ,

  service accounts

  , and GCP security best practices.
• Familiarity with

  monitoring tools

  – Stackdriver, Dataflow job metrics, BQ Query Plans.