Developer - PySpark

Compunnel

2 - 7 years

6 - 10 Lacs

noida

Posted:2 weeks ago| Platform:

Apply

Skills Required

pyspark spark debugging python data manipulation hive continuous integration kubernetes ci/cd sql docker postgresql gcp mysql hadoop bigquery etl azure data lake microsoft azure data engineering dataproc pipeline data bricks troubleshooting aws

Work Mode

Work from Office

Job Type

Full Time

Job Description

We are looking for skilled PySparkDevelopers to design and develop scalable data processing solutions. Therole involves working with big data platforms, building ETL pipelines,and collaborating with cross-functional teams to ensure data availability,performance, and quality.

Qualification:BE / B. TechLocation:Pune,

Key Responsibilities

Design, develop, and optimize ETL/ELT pipelines using PySpark.
Ingest, clean, transform, and process structured/semi-structured/unstructured data.
Work with large-scale datasets on distributed computing platforms (Hadoop, Spark, Databricks).
Integrate data from multiple sources including Delta Lake, Data Lake, RDBMS, APIs.
Ensure high-performance, scalability, and reliability of data pipelines.
Collaborate with data engineers, analysts, and data scientists to deliver business-ready datasets.
Implement monitoring, logging, and error-handling for data pipelines.
Contribute to data quality frameworks and best practices

Required Skills & Experience

Bachelors degree in computer science, computer engineering or similar.
Excellent communication skills.
2-10 years of experience in data engineering with strong hands-on expertise in PySpark.
Proficiency in Python programming for data manipulation and pipeline development.
Strong knowledge of Spark SQL, DataFrames, and RDDs.
Experience with big data ecosystems Hadoop, Hive, or Databricks.
Good understanding of SQL and relational databases (PostgreSQL, MySQL, Oracle).
Familiarity with cloud platforms (AWS, Azure, GCP) and services (S3, ADLS, BigQuery).
Experience with version control (Git) and CI/CD tools.
Strong debugging, performance optimization, and troubleshooting skills.
Experience with containerization (Docker, Kubernetes).

More Jobs at Compunnel

Trainer (Leaning and Development)

Noida

1 - 4 yrs

INR 4 - 6 Lacs

Hiring For Bench Sales Recruiter - Pune_Onsite

Pune

2 - 7 yrs

INR 3 - 7 Lacs

Senior ServiceNow Designer/Architect(Remote)

Noida

10 - 15 yrs

INR 35 - 60 Lacs

Bench Sales- Recruiter/ Senior Recruiter/ Lead Recruiter

Noida, Bhopal, Hyderabad

3 - 8 yrs

INR 4 - 8 Lacs

Us It Recruiter

Noida, Bhopal, Jaipur

1 - 4 yrs

INR 3 - 7 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.