MLSE (Python/Pyspark)

You should have 6-8 years of hands-on exposure with Big Data technologies such as pySpark (Data frame and SparkSQL), Hadoop, and Hive. In addition, you should possess good experience in Python and Bash Scripts, along with a solid understanding of SQL and data warehouse concepts. Strong analytical, problem-solving, data analysis, and research skills are essential for this role. You must also demonstrate the ability to think innovatively and not rely solely on existing tools. Excellent communication, presentation, and interpersonal skills are a must. Experience with Cloud Platform provided Big Data technologies like IAM, Glue, EMR, RedShift, S3, and Kinesis is required. Knowledge of orchestration with Airflow and any job scheduler experience is preferred. Experience in migrating workloads from on-premise to cloud and cloud to cloud migrations would be a plus. The ideal candidate should be proficient in Python, pyspark, and SQL. As a part of the role, you will be responsible for developing efficient ETL pipelines based on business requirements while adhering to development standards and best practices. You will also perform integration testing of different pipelines in the AWS environment and provide estimates for development, testing, and deployments on various environments. Participation in code peer reviews to ensure compliance with best practices is essential. Moreover, creating cost-effective AWS pipelines with necessary AWS services like S3, IAM, Glue, EMR, Redshift, etc., is a key responsibility. Overall, you should have 6 to 8 years of relevant experience in this field to excel in this position. Job Reference Number: 13024,