Spark Developer

As an ETL Developer at our company, your responsibilities will include designing, implementing, and optimizing distributed data processing jobs to handle large-scale data in Hadoop Distributed File System (HDFS) using Apache Spark and Python. You will need a deep understanding of data engineering principles, proficiency in Python, and hands-on experience with Spark and Hadoop ecosystems. Collaboration with data engineers, analysts, and business stakeholders to process, transform, and drive insights and data-driven decisions will also be a key aspect of your role. Key Responsibilities: - Design and implement Spark applications to process and transform large datasets in HDFS. - Develop ETL Pipelines in Spark using Python for data ingestion, cleaning, aggregation, and transformations. - Optimize Spark jobs for efficiency, reducing run time and resource usage. - Fine-tune memory management, caching, and partitioning strategies for optimal performance. - Load data from different sources into HDFS, ensuring data accuracy and integrity. - Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop, etc. - Troubleshoot and debug Spark Job failures, monitor job logs, and Spark UI to identify issues. Qualifications: - 2-5 years of relevant experience - Experience in programming/debugging used in business applications - Working knowledge of industry practice and standards - Comprehensive knowledge of the specific business area for application development - Working knowledge of program languages - Consistently demonstrates clear and concise written and verbal communication - Expertise in handling complex large-scale Warehouse environments - Hands-on experience writing complex SQL queries, exporting and importing large amounts of data using utilities Education: - Bachelor's degree in a quantitative field (such as Engineering, Computer Science) or equivalent experience This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required.,