Data Engineer

Key Responsibilities
Designed and developed scalable ETL pipelines using Cloud Functions, Cloud Dataproc (Spark), and BigQuery as the central data warehouse for large-scale batch and transformation workloads.Implemented efficient data modeling techniques in BigQuery (including star/snowflake schemas, partitioning, and clustering) to support high-performance analytics and reduce query costs.Built end-to-end ingestion frameworks leveraging Cloud Pub/Sub and Cloud Functions for real-time and event-driven data capture.Used Apache Airflow (Cloud Composer) for orchestration of complex data workflows and dependency management.Applied Cloud Data Fusion and Datastream selectively for integrating specific sources (e.g., databases and legacy systems) into the pipeline.Developed strong backtracking and troubleshooting workflows to quickly identify data issues, job failures, and pipeline bottlenecks, ensuring consistent data delivery and SLA compliance.Integrated robust monitoring, alerting, and logging to ensure data quality, integrity, and observability.Tech stackGCP: BigQuery, Cloud Functions, Cloud Dataproc (Spark), Pub/Sub, Data Fusion, DatastreamOrchestration: Apache Airflow (Cloud Composer)Languages: Python, SQL, PySparkConcepts: Data Modeling, ETL/ELT, Streaming & Batch Processing, Schema Management, Monitoring & LoggingSome of the most important data sources: (need to know ingestion technique on these)CRM Systems (cloud-based and internal)SalesforceTeradataMySQLAPIOther 3rd-party and internal operational systemsSkills: etl/elt,cloud data fusion,schema management,sql,pyspark,cloud dataproc (spark),monitoring & logging,data modeling,bigquery,etl,cloud pub/sub,python,gcp,bigquerry,streaming & batch processing,datastream,cloud functions,spark,apache airflow (cloud composer)