Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Home
>
Jobs in pune
>
Pylon Management Consulting
>
Data Engineer- Pyspark

Data Engineer- Pyspark

Pylon Management Consulting

8 - 12 years

15 - 25 Lacs

pune

Posted:None| Platform:

Apply

Skills Required

pyspark data engineering data bricks

Work Mode

Hybrid

Job Type

Full Time

Job Description

Data Engineer Job Description:

Job Responsibilities:

Design and implement highly scalable, distributed data pipelines using Spark and Delta

Lake.

Collaborate with the team to integrate Spark workflows into orchestration tools like

Airflow.

Implement best practices for job scheduling, resource allocation, and failure recovery in

Apache Spark.

Conduct performance benchmarking and recommend improvements for Apache Spark

tuning based on workload patterns.

Monitor cluster health and troubleshoot issues to maintain high availability and stability.
Work closely with the data engineering team to define data models and transformations

using DBT Core and Spark SQL.

Required Skills:

hands-on experience in building Big Data enterprise grade data platforms.
Proven track record of optimizing Apache Spark clusters for large-scale workloads and

complex job patterns.

Strong proficiency in Apache Spark Core, Spark SQL, and PySpark.
Experience with Delta Lake for managing data lakes and supporting ACID transactions.
Familiarity with Debezium with Apache Kafka, Airflow, and DBT Core is a plus.
Strong understanding of distributed systems, memory management, and resource

allocation strategies.

Solid experience with on-premises infrastructure and ability to collaborate with system

administrators for cluster setup and tuning.

Problem-solving mindset and ability to work independently.

Nice to Have:

Experience with Open Metadata or similar data governance and discovery tools

More Jobs at Pylon Management Consulting

Senior Associate Level 2

Chennai, Bengaluru, Gurgaon

7 - 10 yrs

INR 20 - 35 Lacs

Non It Recruiter

Bengaluru

3 - 7 yrs

INR 4 - 6 Lacs

Analyst Tester (Manual Testing) | Leading Fintech MNC

Gurugram

1 - 5 yrs

INR 7 - 15 Lacs

Technical Recruiter

Bengaluru

2 - 7 yrs

INR 3 - 7 Lacs

Manager/AVP/VP-Real Estate Acquisition: Mumbai & Bangalore (PropTech)

Bengaluru, Mumbai (All Areas)

5 - 10 yrs

INR 20 - 30 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.