Spark Developer

As an ETL Developer at our company, you will be responsible for designing, implementing, and optimizing distributed data processing jobs to handle large-scale data in Hadoop Distributed File System (HDFS) using Apache Spark and Python. Your role will require a deep understanding of data engineering principles, proficiency in Python, and hands-on experience with Spark and Hadoop ecosystems. You will collaborate with data engineers, analysts, and business stakeholders to process, transform, and drive insights for data-driven decisions. **Key Responsibilities:** - Design and Implement Spark applications to process and transform large datasets in HDFS. - Develop ETL Pipelines in Spark using Python for data Ingestion, cleaning, aggregation, and transformations. - Optimize Spark jobs for efficiency, reducing run time and resource usage. - Finetune memory management, caching, and partitioning strategies for optimal performance. - Load data from different sources into HDFS, ensuring data accuracy and integrity. - Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop, etc. - Troubleshoot and debug Spark Job failures, monitor job logs, and Spark UI to identify issues. **Qualifications:** - 2-5 years of relevant experience. - Experience in programming/debugging used in business applications. - Working knowledge of industry practice and standards. - Comprehensive knowledge of the specific business area for application development. - Working knowledge of program languages. - Consistently demonstrates clear and concise written and verbal communication. - Expertise in handling complex large-scale Warehouse environments. - Hands-on experience writing complex SQL queries, exporting and importing large amounts of data using utilities. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity, review Accessibility at Citi.,