Data Engineer

Location:

No-of Positions :

JD :

• Hands-on expertise in

  GCP

  data stack:

  BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow)

  .
• Strong

  Spark

  (PySpark or Scala) for batch processing on Dataproc.
• Solid

  Airflow

  DAG design (idempotent tasks, backfills, retries, SLAs).
• Advanced

  SQL

  and data modeling (star/snowflake, slowly changing dimensions, partition strategies).
• Proficiency in

  Python

  (preferred) or

  Scala/Java

  for data engineering.
• Experience with

  Git

  and CI/CD (Cloud Build/GitHub Actions/GitLab CI).
• Familiarity with

  security & governance

  on GCP (IAM, service accounts, secrets management, VPC-SC basics).
• Strong debugging skills and ownership mindset; clear communication with technical and non-technical stakeholders.

  Key Responsibilities

• Design, develop, and maintain ELT/ETL pipelines on GCP using

  Dataflow/Beam

  ,

  Dataproc/Spark

  , and

  Airflow/Composer

  .
• Model and optimize datasets in

  BigQuery

  (partitioning, clustering, materialized views, UDFs).
• Build streaming and near-real-time ingestion using

  Pub/Sub

  , Dataflow, and CDC where applicable.
• Implement data quality checks, validation frameworks, and SLAs; monitor pipelines via Cloud Monitoring/Logging.
• Optimize performance and cost across GCS, Dataproc autoscaling, and BigQuery slot usage.
• Contribute to standards for coding, CI/CD, observability, and documentation; perform code reviews.
• Partner with Analytics/BI/ML teams to productize datasets and ensure strong data contracts.
• Support production operations, including on-call rotations for critical pipelines (as needed).

  Good-to-Have

• Snowflake

  (migrations to/from BigQuery, performance tuning, tasks/streams).

• dbt

  for transformations and testing;

  Great Expectations

  or similar DQ frameworks.

• Terraform

  for infrastructure as code on GCP.

• Kafka

  or additional streaming tooling;

  Cloud Run/Functions

  for glue services.
• BI exposure (

  Looker/Looker Studio

  , Tableau, or Power BI).
• GCP

  Professional Data Engineer

  certification.