PySpark Developer

Description

Position :

PySpark Developer

Work Location :

Navi Mumbai (Work from Office)

Experience :

3 to 4 Years

Notice Period :

Immediate to 15 Days

About The Role

We are looking for a highly skilled PySpark Developer with strong experience in Python, SQL, and AWS cloud environments to design, build, and maintain scalable data pipelines and automation workflows. The ideal candidate should be comfortable handling large datasets, integrating APIs, and optimizing complex queries for performance.This role offers an exciting opportunity to work in a fast-paced environment with exposure to modern data platforms, AWS services, and big data tools.

Key Responsibilities

• Design, develop, and maintain data integration and transformation pipelines using PySpark and Python.
• Build modular, reusable, and efficient scripts to process data from multiple sources such as APIs, databases, SFTP, and AWS S3.
• Handle and transform various data formats including CSV, JSON, XML, Excel, and Parquet.
• Write and optimize complex SQL queries for efficient data extraction, transformation, and loading (ETL/ELT).
• Work extensively in AWS cloud environments using services such as Glue, S3, CloudWatch, EC2, RDS, and CLI.
• Implement data validation, cleansing, and audit logging processes to ensure data quality and consistency.
• Automate recurring data tasks, jobs, and reporting workflows to improve system efficiency.
• Collaborate with the data engineering and analytics teams to design and apply Data Lake and Data Warehouse architectures.
• Monitor and troubleshoot data processes, identify performance bottlenecks, and optimize workflows.
• Maintain clear documentation for data pipelines, transformations, and automation scripts.
  

Mandatory Skills

• Strong proficiency in PySpark and Python scripting (including libraries like Pandas and NumPy).
• Expertise in SQL (writing and optimizing complex queries).
• Hands-on experience with AWS Cloud Services especially Glue, S3, CloudWatch, and AWS CLI.
• Understanding of data integration, transformation, and processing techniques.
  

Good-to-Have Skills

• Basic knowledge of data modelling and schema design.
• Experience with ETL/ELT concepts and tools.
• Familiarity with Data Warehousing and Data Lake architectures.
• Working knowledge of Shell scripting for automation.
• Experience in API integration and data ingestion.
• Exposure to Databricks or similar data processing environments.
  

Why Join Us

• Opportunity to work on cutting-edge cloud and big data technologies.
• Exposure to end-to-end data engineering projects.
• Collaborative environment with strong learning and career growth potential.
• Immediate onboarding for suitable candidates.
  

(ref:hirist.tech)