Job Description

Senior PySpark Data Engineer Location: Pune/Hybrid Experience: 5-8 years Notice Period: Immediate to 15 days Mandatory Skills: Python SQL ETL Informatica PowerCenter AWS/Azure Good to Have: IDMC Tech Stack Table Skills Experience Rating out of 10 Python SQL ETL Informatica PowerCenter Aws/Azure Job Summary: We are seeking a Senior PySpark Data Engineer with extensive experience in developing, optimizing, and maintaining data processing jobs using PySpark. The ideal candidate will possess a robust background in SQL and ETL processes, along with proficiency in cloud platforms such as AWS or Azure. This role will require excellent analytical skills and the ability to communicate effectively with both technical and non-technical stakeholders. Key Responsibilities: Design, develop, and optimize PySpark jobs for enhanced performance and scalability. Collaborate with data architects and business analysts to understand data requirements and translate them into technical specifications. Redesign and maintain complex SQL queries and stored procedures to support data extraction and transformation processes. Utilize ETL tools, specifically Informatica PowerCenter, to build effective data pipelines. Troubleshoot and resolve data quality issues and performance bottlenecks. Mentor and provide technical guidance to a team of developers to enhance productivity and code quality. Stay updated with new technologies and practices to continually improve data processing capabilities. Qualifications: Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field. Experience: 5-8 years of experience in data engineering, with a strong focus on PySpark and ETL processes. Technical Skills: Must-Have: Extensive experience with PySpark, focusing on job optimization techniques. Proficiency in SQL, with experience in SQL Server, MySQL, or other relational databases. Strong knowledge of ETL concepts and tools, particularly Informatica PowerCenter and IDMC. Excellent analytical and troubleshooting skills. Strong communication skills for effective collaboration. Good to Have: Basic knowledge of Unix commands and Shell scripting. Experience in leading and mentoring development teams. Familiarity with Azure/Fabric. Show more Show less