Spark Developer

As an ETL Developer at our company, you will be responsible for designing, implementing, and optimizing distributed data processing jobs to handle large-scale data in Hadoop Distributed File System(HDFS) using Apache Spark and Python. Your role will require a deep understanding of data engineering principles, proficiency in Python, and hands-on experience with Spark and Hadoop ecosystems. You will collaborate with data engineers, analysts, and business stakeholders to process, transform, and drive insights and data-driven decisions. **Responsibilities:** - Design and Implement of Spark applications to process and transform large datasets in HDFS. - Develop ETL Pipelines in Spark using Python for data ingestion, cleaning, aggregation, and transformations. - Optimize Spark jobs for efficiency, reducing run time and resource usage. - Fine-tune memory management, caching, and partitioning strategies for optimal performance. - Load data from different sources into HDFS, ensuring data accuracy and integrity. - Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop, etc. - Troubleshoot and debug Spark Job failures, monitor job logs, and Spark UI to identify issues. **Qualifications:** - 2-5 years of relevant experience. - Experience in programming/debugging used in business applications. - Working knowledge of industry practice and standards. - Comprehensive knowledge of specific business areas for application development. - Working knowledge of program languages. - Consistently demonstrates clear and concise written and verbal communication. - Expertise in handling complex large-scale Warehouse environments. - Hands-on experience writing complex SQL queries, exporting and importing large amounts of data using utilities. **Education:** - Bachelor's degree in a quantitative field (such as Engineering, Computer Science) or equivalent experience. This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required.,