Pyspark Developer

Required Skills:

• Python Programming:

  Strong ability to write clean and efficient code.

• Spark SQL:

  Good understanding of Spark SQL for distributed data processing.

• Data Processing:

  Experience with large datasets and structured data manipulation.

• SQL Fundamentals:

  Ability to write queries and optimize database performance.

• Problem-Solving:

  Analytical mindset to debug and optimize workflows.

Preferred Skills:

• AWS Cloud Services:

  Familiarity with

  S3, Redshift, Lambda, EMR

  is an advantage.

• ETL Development:

  Understanding of ETL processes and data engineering principles.

• Version Control:

  Experience using

  Git

  for collaborative development.

• Big Data Tools:

  Exposure to

  Hive, PySpark

  , or similar technologies.

Roles & Responsibilities

• Develop and optimize

  Python scripts

  for data processing and automation.
• Write efficient

  Spark SQL queries

  for handling large-scale structured data.
• Assist in

  ETL pipeline development

  and maintenance.
• Support

  data validation

  and integrity checks across systems.
• Collaborate with teams to implement

  cloud-based solutions

  (AWS preferred).
• Optimize performance of data queries and workflows.
• Troubleshoot and debug issues in existing applications.
• Document processes and ensure best practices in coding and data handling.