Associate Principal - Architecture (Big Data)

Role Overview: You will be responsible for utilizing your strong experience in PySpark, Python, and Unix scripting to work on data processing tasks. Your expertise in SparkSQL and Hive will be essential in writing SQLs and creating views. Additionally, your excellent communication skills will be valuable in collaborating with team members. Knowledge of the Insurance domain is a plus, along with a good understanding of the Hadoop Ecosystem and Architecture, including HDFS, Map Reduce, Pig, Hive, Oozie, and Yarn. You will also need familiarity with AWS services such as Glue, AWS S3, Lambda function, Step Function, and EC2. Data migration exposure from platforms like Hive/S3 to Data Bricks will be part of your responsibilities. Your ability to prioritize, plan, organize, and manage multiple tasks efficiently while maintaining high-quality work is crucial for success in this role. Key Responsibilities: - Utilize strong experience in PySpark, Python, and Unix scripting for data processing tasks - Write SQLs, create views, and work with SparkSQL and Hive - Collaborate effectively with team members using excellent oral and written communication skills - Demonstrate knowledge of the Insurance domain and the Hadoop Ecosystem and Architecture - Use AWS services like Glue, AWS S3, Lambda function, Step Function, and EC2 - Perform data migration from platforms like Hive/S3 to Data Bricks Qualifications Required: - Technical experience of 6-8 years in Pyspark, AWS (Glue, EMR, Lambda & Steps functions, S3) - 3+ years of experience in Bigdata/ETL with Python+Spark+Hive and 3+ years of experience in AWS - Proficiency in Pyspark, AWS (Glue, EMR, Lambda & Steps functions, S3), and Big data with Python+Spark+Hive experience - Exposure to big data migration Additional Company Details: There are no additional details about the company mentioned in the job description.,