Senior AWS Data Engineer

Job Description

Senior AWS Data Engineer  Location: Remote Experience: 7–8 years Open Positions: 4 Work Hours: Must be comfortable working in UK shift timings Job Summary: We are looking for experienced AWS Data Engineers with a strong foundation in data engineering principles and hands-on expertise in building scalable data pipelines and assets. The ideal candidate should have in-depth experience in Python, PySpark, and AWS services, with a focus on performance optimization and modern software engineering practices. Key Responsibilities: Design, develop, and optimize data pipelines using PySpark and Spark SQL. Refactor legacy codebases to improve clarity, maintainability, and performance. Apply test-driven development (TDD) principles, writing unit tests to ensure robust and reliable code. Debug complex issues, including performance bottlenecks, concurrency challenges, and logical errors. Utilize Python best practices, libraries, and frameworks effectively. Implement and manage code versioning and artifact repositories (Git, JFrog Artifactory). Work with various AWS services, including S3, EC2, Lambda, Redshift, CloudFormation, etc. Provide architectural insights and articulate the benefits and trade-offs of AWS components. Requirements: 7–8 years of strong hands-on experience with PySpark and Python (including Boto3 and relevant libraries). Proven ability to write clean, optimized, and scalable code in Spark SQL and PySpark. Solid understanding of version control systems, preferably Git, and experience with JFrog Artifactory. Strong knowledge of AWS services and architecture; ability to explain service benefits and use cases. Experience in modernizing legacy systems and implementing clean code practices. Strong debugging skills with the ability to identify and fix complex bugs and performance issues. Familiarity with TDD and writing comprehensive unit tests