Big Data Developer

Role

Location

Experience

Key Skills Required

• Google Cloud Platform (GCP)
• BigQuery (BQ)
• Dataflow
• Dataproc
• Cloud Spanner
• Strong knowledge of distributed systems, data processing frameworks, and big data architecture.
• Proficiency in programming languages like Python, Java, or Scala.
  

Roles And Responsibilities

• BigQuery (BQ):
• Design and develop scalable data warehouses using BigQuery.
• Optimize SQL queries for performance and cost-efficiency in BigQuery.
• Implement data partitioning and clustering strategies.
• Dataflow:
• Build and maintain batch and streaming data pipelines using Apache Beam on GCP Dataflow.
• Ensure data transformation, enrichment, and cleansing as per business needs.
• Monitor and troubleshoot pipeline performance issues.
• Dataproc:
• Develop and manage Spark and Hadoop jobs on GCP Dataproc.
• Perform ETL/ELT operations using PySpark, Hive, or other tools.
• Automate and orchestrate jobs for scheduled data workflows.
• Cloud Spanner:
• Design and manage globally distributed, scalable transactional databases using Cloud Spanner.
• Optimize schema and query design for performance and reliability.
• Implement high availability and disaster recovery strategies.
• General Responsibilities:
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Implement data quality and data governance best practices.
• Ensure security and compliance with GCP data handling standards.
• Participate in code reviews, CI/CD deployments, and Agile development cycles.