Jobs
Interviews
UST logo

GCP & PySpark with ETL - Lead

5 - 9 years

0 Lacs

kochi kerala

Posted:2 days ago| Platform: Shine logo

Apply

Skills Required

cloud storage python sql kafka docker kubernetes devops pyspark google cloud platform gcp bigquery cloud dataflow cloud composer apache airflow cicd pipelines pubsub terraform monitoring tools prometheus stackdriver datadog

Work Mode

On-site

Job Type

Full Time

Job Description

As a Data Engineer at our company, you will be responsible for designing, developing, and optimizing ETL pipelines using PySpark on Google Cloud Platform (GCP). Your role will involve working with various GCP services such as BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration. Key Responsibilities: - Develop and optimize Spark-based ETL processes for large-scale data processing - Implement best practices for data governance, security, and monitoring in a cloud environment - Collaborate with data engineers, analysts, and business stakeholders to understand data requirements - Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution - Automate data workflows using Apache Airflow or Cloud Composer - Ensure data quality, validation, and consistency across pipelines Qualifications Required: - At least 5 years of experience in ETL development with a focus on PySpark - Strong hands-on experience with GCP services such as BigQuery, Cloud Dataflow/Apache Beam, Cloud Composer, and Cloud Storage - Proficiency in Python and PySpark for big data processing - Experience with data lake architectures, data warehousing concepts, and SQL for data querying and transformation - Knowledge of CI/CD pipelines for data pipeline automation - Strong debugging and problem-solving skills - Familiarity with Kafka or Pub/Sub for real-time data processing - Knowledge of Terraform for infrastructure automation on GCP - Experience with containerization (Docker, Kubernetes) - Familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog Your skills in GCP, PySpark, and ETL will be crucial as you contribute to the development and optimization of data pipelines in a cloud environment.,

More Jobs at UST

Architect II - Technical SAP ABAP

Bengaluru

12 - 15 yrs
|

INR 0 - 0 Lacs

Lead I - SAP FICO with PS (Project Systems) and RAR

Bengaluru

5 - 7 yrs
|

INR 0 - 0 Lacs

Lead II - SAP ABAP

Bengaluru

7 - 9 yrs
|

INR 0 - 0 Lacs

Procurement

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Manual & Python Automation

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
UST logo
UST

IT Services and IT Consulting

Aliso Viejo CA

RecommendedJobs for You

UST logo

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

Kochi, Kerala, India

UST logo

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

Kochi, Kerala, India