Job
Description
Role Overview: As a skilled Data Engineer with experience in Google Cloud Platform (GCP), PySpark, SQL, and ETL processes, you will be responsible for building, optimizing, and maintaining scalable data pipelines and workflows. Your expertise in technologies such as Apache Airflow, PySpark, and cloud-native tools will be crucial in ensuring efficient data processing. Key Responsibilities: - Design, develop, and maintain efficient and scalable data pipelines using PySpark and SQL. - Build and manage workflows/orchestration using Apache Airflow. - Utilize GCP services like BigQuery, Cloud Storage, Dataflow, and Composer. - Implement and optimize ETL processes to guarantee data quality, consistency, and reliability. - Collaborate with data analysts, data scientists, and engineering teams to meet data requirements. - Monitor pipeline performance and troubleshoot issues promptly. - Write clean, maintainable, and well-documented code. Qualifications Required: - Strong programming skills in PySpark and SQL. - Proven experience with Google Cloud Platform (GCP). - Solid grasp of ETL concepts and data pipeline design. - Hands-on experience with Apache Airflow or similar orchestration tools. - Familiarity with cloud-based data warehousing and storage systems such as BigQuery and Cloud Storage. - Experience in performance tuning and optimizing large-scale data pipelines. - Strong problem-solving skills and the ability to work both independently and collaboratively within a team. Role Overview: As a skilled Data Engineer with experience in Google Cloud Platform (GCP), PySpark, SQL, and ETL processes, you will be responsible for building, optimizing, and maintaining scalable data pipelines and workflows. Your expertise in technologies such as Apache Airflow, PySpark, and cloud-native tools will be crucial in ensuring efficient data processing. Key Responsibilities: - Design, develop, and maintain efficient and scalable data pipelines using PySpark and SQL. - Build and manage workflows/orchestration using Apache Airflow. - Utilize GCP services like BigQuery, Cloud Storage, Dataflow, and Composer. - Implement and optimize ETL processes to guarantee data quality, consistency, and reliability. - Collaborate with data analysts, data scientists, and engineering teams to meet data requirements. - Monitor pipeline performance and troubleshoot issues promptly. - Write clean, maintainable, and well-documented code. Qualifications Required: - Strong programming skills in PySpark and SQL. - Proven experience with Google Cloud Platform (GCP). - Solid grasp of ETL concepts and data pipeline design. - Hands-on experience with Apache Airflow or similar orchestration tools. - Familiarity with cloud-based data warehousing and storage systems such as BigQuery and Cloud Storage. - Experience in performance tuning and optimizing large-scale data pipelines. - Strong problem-solving skills and the ability to work both independently and collaboratively within a team.
