AWS Pyspark Developer

Job Summary

AWS PySpark Developer

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark.
• Build and maintain data ingestion frameworks on AWS.
• Develop batch and real-time data processing solutions.
• Optimize Spark jobs for performance and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Develop data lake and data warehouse solutions.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with business stakeholders, architects, and analytics teams.
• Support production deployments and troubleshoot critical issues.
• Follow DevOps and CI/CD best practices.

Mandatory Technical Skills

AWS Services

• Amazon S3
• AWS Glue
• AWS EMR
• AWS Lambda
• Amazon Redshift
• AWS IAM
• CloudWatch
• AWS Step Functions

Big Data & Data Engineering

• PySpark
• Apache Spark
• Spark SQL
• DataFrames & RDDs
• ETL/ELT Development
• Data Modeling
• Data Warehousing Concepts

Programming

• Python
• SQL
• Shell Scripting

Database Technologies

• Redshift
• PostgreSQL
• MySQL
• DynamoDB (Preferred)

DevOps & CI/CD

• Git
• Jenkins / AWS CodePipeline
• Docker
• Terraform (Preferred)

Good-to-Have Skills

• Databricks
• Kafka
• Apache Airflow
• Delta Lake
• Snowflake
• Kubernetes
• Streaming Technologies
• Machine Learning Exposure