Sr PySpark Developer (Bigdata)

We are seeking a highly skilled PySpark Developer with expertise in Distributed data processing to optimize Spark Jobs and ensure efficient data processing in a Big Data platform. This role demands a deep understanding of Spark performance tuning, distributed computing, and Big data architecture. Responsibilities include creating data tools for analytics and data scientist team members, collaborating with data and analytics experts to enhance functionality in the data system, and having the ability to adjust priorities quickly based on circumstances. The ideal candidate should have 8+ years of relevant experience in Apps Development or systems analysis. Key Responsibilities: - Analyze and comprehend existing data ingestion and reconciliation frameworks - Develop PySpark programs to process large datasets in Hive tables and Big data platforms - Perform complex transformations and advanced data manipulations - Fine-tune Spark jobs for performance optimization at scale - Work closely with Data Engineers, Architects, and Analysts to understand data reconciliation requirements - Collaborate with cross-functional teams to enhance data ingestion, transformation, and validation workflows Required Skills and Qualifications: - Extensive hands-on experience with Python, PySpark, and PyMongo for efficient data processing across distributed and columnar databases - Expertise in Spark Optimization techniques, debugging Spark performance issues, and optimizing resource utilization - Proficiency in Python and Spark DataFrame API, with experience in complex data transformations using PySpark - Experience with large-scale distributed data processing, and understanding of Big Data architecture and distributed computing frameworks - Strong problem-solving and analytical skills - Experience with CI/CD for data pipelines - Experience with SnowFlake for data processing and integration Education: - Bachelors degree/University degree in Computer Science or equivalent experience - Masters degree preferred Citi is an equal opportunity and affirmative action employer, encouraging all qualified interested applicants to apply for career opportunities. For applicants with disabilities requiring accommodation, review Accessibility at Citi.,