Aws Data Engineer

Job Summary:

We are looking for a Senior Data Engineer with 6+ years of experience in designing, developing, and supporting scalable data pipelines. The candidate should have strong hands-on experience in Python, PySpark, AWS data services, Spark, EMR, Delta Lake, Kinesis, and workflow orchestration.

The role involves building and maintaining ETL/ELT pipelines, working with large datasets, optimizing data processing, and quickly resolving data-related issues.

Key Responsibilities:

• Design, develop, and maintain scalable and reliable data pipelines using Python and PySpark.
• Build and manage ETL/ELT processes for large and complex datasets.
• Develop data processing solutions using Apache Spark and AWS EMR.
• Implement serverless data workflows using AWS Lambda and Step Functions.
• Work with AWS S3, Glue, Athena, Redshift, and other AWS data services.
• Build and maintain real-time and streaming data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Work with Delta Lake for reliable data storage and processing.
• Perform data modelling, data transformation, and performance optimization.
• Monitor data pipelines and troubleshoot failures and data quality issues.
• Handle and resolve data hydration-related issues quickly and effectively.
• Implement best practices for code quality, version control, and deployment.
• Work with Git and Bitbucket for source code management and CI/CD processes.
• Collaborate with cross-functional teams to understand requirements and deliver data solutions on time.
• Support production data pipelines and resolve issues within agreed timelines.

Mandatory Skills:

• 6+ years of experience in Data Engineering.
• Strong hands-on experience with Python.
• Strong experience with PySpark and Apache Spark.
• Hands-on experience with AWS EMR.
• Hands-on experience with AWS Lambda.
• Hands-on experience with AWS Step Functions.
• Experience with Delta Lake.
• Experience with Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Strong understanding of ETL/ELT processes and data pipelines.
• Experience working with large datasets and distributed data processing.
• Strong troubleshooting and production support skills.
• Experience with AWS S3 and other AWS data services.

Preferred Skills:

• Experience with AWS Glue, Athena, and Redshift.
• Experience with Snowflake and cloud-based data warehousing.
• Knowledge of data modelling and performance optimization.
• Experience with CI/CD processes.
• Hands-on experience with Git and Bitbucket.
• Experience in monitoring and troubleshooting production data pipelines.
• Good understanding of data quality and data validation processes.
• Strong communication and collaboration skills.
• Ability to work independently and deliver solutions within tight timelines.