Manager & Deputy Manager Pyspark Developer

As a skilled PySpark Developer with 4-5 or 2-3 years of experience, you will be responsible for developing and maintaining data processing pipelines using PySpark, Apache Spark's Python API. You will collaborate closely with data engineers, data scientists, and stakeholders to design and implement scalable and efficient data processing solutions. Key Responsibilities: - Designing, developing, and maintaining PySpark data processing pipelines to handle large volumes of structured and unstructured data - Collaborating with data engineers and data scientists to understand data requirements and create efficient data models and transformations - Optimizing and tuning PySpark jobs for improved performance, scalability, and reliability - Implementing data quality checks, error handling, and monitoring mechanisms to ensure data accuracy and pipeline robustness - Developing and managing documentation for PySpark code, data pipelines, and data workflows Qualifications Required: - Bachelor's or Master's degree in Computer Science, Data Science, or a related field - Strong expertise in the Big Data ecosystem including Spark, Hive, Sqoop, HDFS, Map Reduce, Oozie, Yarn, HBase, Nifi - Experience in developing production-ready Spark applications using Spark RDD APIs, Data frames, Datasets, Spark SQL, and Spark Streaming - Strong experience in HIVE Bucketing and Partitioning, and writing complex hive queries using analytical functions - Knowledge of writing custom UDFs in Hive to support custom business requirements If you meet the qualifications mentioned above and are interested in this position, please email your resume to careers@cdslindia.com, mentioning the position applied for in the subject column.,