Data Engineer(Spark)

You have a deep experience in developing data processing tasks using PySpark/spark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations. Your responsibilities will include developing, programming, and maintaining applications using the Apache Spark and python open-source framework. You will work with different aspects of the Spark ecosystem, including Spark SQL, DataFrames, Datasets, and streaming. As a Spark Developer, you must have strong programming skills in Python, Java, or Scala. It is essential that you are familiar with big data processing tools and techniques, have a good understanding of distributed systems, and possess proven experience as a Spark Developer or a related role. Your problem-solving and analytical thinking skills should be excellent. Experience with building APIs for provisioning data to downstream systems is required. Working experience on any Cloud technology like AWS, Azure, Google is an added advantage. Hands-on experience with AWS S3 Filesystem operations will be beneficial for this role.,