Date Engineer(Python, Pyspark and AWS)

Key Skills Required-

• Minimum 6 years of working experience in Python, Pyspark, AWS and SQL.
• Programming & Frameworks: Python, PySpark
• Databases & Querying: Strong SQL (query optimization, joins, window functions)
• Big Data & Processing: PySpark for distributed data processing, ETL pipelines
• Cloud Platform: AWS (S3, Glue, Lambda, EMR, Redshift, Athena)
• Data Engineering: Data pipeline design, data ingestion, transformation, and integration
• Other Nice-to-Have: Airflow (or other orchestration tools), CI/CD, Docker
  

Responsibilities

• Design and develop scalable ETL pipelines using PySpark and SQL
• Manage data ingestion & transformation on AWS cloud (S3, Glue, EMR, Redshift)
• Optimize SQL queries for performance in large-scale datasets
• Collaborate with data scientists and analysts for data availability & quality
• Implement best practices in data engineering (partitioning, schema design, data governance)
• Monitor, debug, and optimize pipeline performance