AWS Data Engineer

As an AWS Data Engineer, you will play a crucial role in designing, developing, and maintaining data pipelines and assets on AWS. Your responsibilities will include optimizing and refactoring legacy PySpark/Spark SQL code for enhanced performance and maintainability. You will be implementing unit testing/Test-Driven Development (TDD) to ensure the delivery of robust, bug-free code. Additionally, you will be debugging and resolving complex performance, concurrency, and logic issues while managing code versioning and repositories using tools like Git and JFrog Artifactory. Leveraging AWS services such as S3, EC2, Lambda, Redshift, and CloudFormation will be essential for creating scalable data solutions. Key Responsibilities: - Design, develop, and maintain data pipelines and assets on AWS. - Optimize and refactor legacy PySpark/Spark SQL code for performance and maintainability. - Implement unit testing/TDD to ensure robust, bug-free code. - Debug and resolve complex performance, concurrency, and logic issues. - Manage code versioning and repositories using Git and JFrog Artifactory. - Leverage AWS services (S3, EC2, Lambda, Redshift, CloudFormation) for scalable data solutions. Qualifications Required: - 7+ years hands-on experience in Python, PySpark, Boto3, and related frameworks/libraries. - Proven expertise in Spark SQL & PySpark optimization. - Strong knowledge of AWS architecture (S3, EC2, Lambda, Redshift, CloudFormation). - Experience in code refactorization for clean, maintainable solutions. - Familiarity with Git, JFrog Artifactory, and modern CI/CD practices. - Strong debugging and problem-solving skills. - Solid understanding of unit testing and TDD methodologies.,