Pyspark Developer

The position is suited for individuals who have strong PySpark programming skills and have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment.

Education and Experience

Education: B.Tech/M.Tech/MCA/MS/MBA

Experience in design and implementation of Big Data systems using PySpark, database migration, transformation and integration solutions for any Data warehousing project.

Required Skills

• Must have excellent knowledge in Apache Spark and Python programming experience

• Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.

• Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred

• Working experience on Cloud technology architecture like AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage

• Understanding of Unix/Linux + Shell Scripting

• Data modelling experience using advanced statistical analysis,unstructured data processing

• Experience with building APIs for provisioning data to downstream systems by leveraging different frameworks.

• Hands on project experience on Jupyter notebook/ Zeppelin/ PyCharm etc. IDEs

• Hands on experience with AWS S3 Filesystem operations

• Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform

• Experience handling CDC operations for huge volume of data

• Should understand and have operating experience with Agile delivery methodologies

• Should have hands-on experience in the following: data validation, writing unit test cases

• Should have experience in integrating PySpark with downstream and upstream applications through a batch/real-time interface

• Should have experience in fine tuning process and troubleshooting performance issues

• Should have demonstrated expertise in development of design documents like HLD, LLD etc.

• Should have experience in leading requirements gathering and developing solution architecture for Data migration/integration initiatives

• Should have experience in handling client interactions at different phases of the projects

• Should have experience in leading a team in a project or a module

• Should be well versed with onsite/offshore model and its challenges

Preferred Skills

• Exposure to any ETL/Reporting tool (Informatica, Jasper, QlikView, Tableau) is desirable

• Exposure to Jenkins or equivalent CICD tool & Git repository is preferred

• Design & Develop AI/ML model using PySpark on cloud environment