Pyspark Developer

Job Description

The position is suited for individuals who have strong PySpark programming skills and have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment. Education and Experience Education: B.Tech/M.Tech/MCA/MS/MBA Experience in design and implementation of Big Data systems using PySpark, database migration, transformation and integration solutions for any Data warehousing project. Required Skills Must have excellent knowledge in Apache Spark and Python programming experience Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred Working experience on Cloud technology architecture like AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage Understanding of Unix/Linux + Shell Scripting Data modelling experience using advanced statistical analysis,unstructured data processing Experience with building APIs for provisioning data to downstream systems by leveraging different frameworks. Hands on project experience on Jupyter notebook/ Zeppelin/ PyCharm etc. IDEs Hands on experience with AWS S3 Filesystem operations Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform Experience handling CDC operations for huge volume of data Should understand and have operating experience with Agile delivery methodologies Should have hands-on experience in the following: data validation, writing unit test cases Should have experience in integrating PySpark with downstream and upstream applications through a batch/real-time interface Should have experience in fine tuning process and troubleshooting performance issues Should have demonstrated expertise in development of design documents like HLD, LLD etc. Should have experience in leading requirements gathering and developing solution architecture for Data migration/integration initiatives Should have experience in handling client interactions at different phases of the projects Should have experience in leading a team in a project or a module Should be well versed with onsite/offshore model and its challenges Preferred Skills Exposure to any ETL/Reporting tool (Informatica, Jasper, QlikView, Tableau) is desirable Exposure to Jenkins or equivalent CICD tool & Git repository is preferred Design & Develop AI/ML model using PySpark on cloud environment