Pyspark Developer

Job Description

Role & responsibilities Must have excellent knowledge in Apache Spark and Python programming experience Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred Working experience on Cloud technology architecture like AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage Understanding of Unix/Linux + Shell Scripting Data modelling experience using advanced statistical analysis,unstructured data processing Experience with building APIs for provisioning data to downstream systems by leveraging different frameworks. Hands on project experience on Jupyter notebook/ Zeppelin/ PyCharm etc. IDEs Hands on experience with AWS S3 Filesystem operations Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform Experience handling CDC operations for huge volume of data Should understand and have operating experience with Agile delivery methodologies Should have hands-on experience in the following: data validation, writing unit test cases Should have experience in integrating PySpark with downstream and upstream applications through a batch/real-time interface Should have experience in fine tuning process and troubleshooting performance issues Should have demonstrated expertise in development of design documents like HLD, LLD etc. Should have experience in leading requirements gathering and developing solution architecture for Data migration/integration initiatives Should have experience in handling client interactions at different phases of the projects Should have experience in leading a team in a project or a module Should be well versed with onsite/offshore model and its challenges.