PySpark Developer_4+Years

PySpark Development Engineer

PySpark Development Engineer

Required Skills

Technical Skills

• Scripting: Proficiency in Python and Shell scripting for automation and custom transformations.
• Minimum 4 Years of Data Engineering: Proven experience in designing and implementing data pipelines and transformations using PySpark.
• PySpark Expertise: Strong hands-on experience with PySpark for distributed data processing and optimization.
• ETL/Data Engineering: Deep understanding of ETL concepts and experience building automated workflows.
• SQL Proficiency: Ability to write and optimize complex SQL queries for large datasets.
• Cloud & Big Data: Experience with AWS services (Glue, S3, EMR) and big data ecosystems.
• CI/CD & Version Control: Familiarity with Git, Jenkins, and automated deployment pipelines.
• Scripting: Proficiency in Python and Shell scripting for automation and custom transformations.
• Data Quality & Validation: Implementing data quality checks and ensuring integrity throughout the pipeline.
  

Soft Skills

• Exceptional problem-solving and analytical skills.
• Excellent time management and organizational abilities in high-pressure environments.
• Strong written and verbal communication skills.
• Detail-oriented with a proactive approach to identifying and resolving issues.