Manager & Deputy Manager Pyspark Developer

As a PySpark Developer at our company, you will be responsible for developing and maintaining data processing pipelines using PySpark, Apache Spark's Python API. You will collaborate with data engineers, data scientists, and other stakeholders to design and implement scalable and efficient data processing solutions. Key Responsibilities: - Designing, developing, and maintaining PySpark data processing pipelines for processing structured and unstructured data - Collaborating with data engineers and data scientists to understand data requirements and design efficient data models and transformations - Optimizing and tuning PySpark jobs for performance, scalability, and reliability - Implementing data quality checks, error handling, and monitoring mechanisms to ensure data accuracy and pipeline robustness - Developing and maintaining documentation for PySpark code, data pipelines, and data workflows Qualifications Required: - Bachelor's or Master's degree in Computer Science, Data Science, or a related field - Strong expertise in the Big Data ecosystem including Spark, Hive, Sqoop, HDFS, Map Reduce, Oozie, Yarn, HBase, Nifi - Experience in developing production-ready Spark applications using Spark RDD APIs, Data frames, Datasets, Spark SQL, and Spark Streaming - Strong experience of HIVE Bucketing and Partitioning, as well as writing complex hive queries using analytical functions - Knowledge in writing custom UDFs in Hive to support custom business requirements If you meet the above qualifications and are interested in this position, please email your resume, mentioning the position applied for in the subject column at: careers@cdslindia.com.,