MLSE (Python/Pyspark)

As a Senior Big Data Engineer, you will play a crucial role in developing efficient ETL pipelines to meet business requirements while adhering to development standards and best practices. Your responsibilities will include integration testing of various pipelines in AWS environment, providing estimates for development, testing, and deployments across different environments, participating in code peer reviews to ensure compliance with best practices, and creating cost-effective AWS pipelines utilizing services like S3, IAM, Glue, EMR, and Redshift. Key Responsibilities: - Develop efficient ETL pipelines based on business requirements - Conduct integration testing of created pipelines in AWS environment - Provide estimates for development, testing, and deployments in various environments - Participate in code peer reviews to ensure compliance with best practices - Create cost-effective AWS pipelines utilizing services such as S3, IAM, Glue, EMR, and Redshift Qualifications Required: - 6-8 years of hands-on experience with Big Data technologies including pySpark (Data frame and SparkSQL), Hadoop, and Hive - Proficiency in Python and Bash Scripts - Strong understanding of SQL and data warehouse concepts - Analytical, problem-solving, data analysis, and research skills - Ability to think creatively and work independently - Excellent communication, presentation, and interpersonal skills - Hands-on experience with Cloud Platform provided Big Data technologies (IAM, Glue, EMR, RedShift, S3, Kinesis) - Experience in orchestration with Airflow and any job scheduler - Experience in migrating workloads from on-premise to cloud and cloud to cloud migrations Good to have skills: - Python, pyspark, SQL In this role, your experience of 6 to 8 years in the field will be valuable as you contribute to the successful execution of Big Data projects.,