Job
Description
Job Title: Data Platform Developer Key Responsibilities As a Data Platform Developer, you will: Solution Design & Development: Design, build, and unit test applications on the Spark framework using Python (PySpark). Translate requirements into full-fledged and scalable PySpark-based applications for both batch and streaming requirements. Data Pipeline Development: Develop and execute data pipeline testing processes, and validate business rules and policies. Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, and HDFS compression codecs. Big Data Ecosystem Management: Apply in-depth knowledge of various Hadoop and NoSQL databases. Automation & CI/CD: Create and maintain integration and regression testing frameworks on Jenkins, integrated with Bitbucket and/or GIT repositories. Agile Collaboration: Participate actively in the Agile development process, documenting and communicating issues and bugs related to data standards in scrum meetings. Work collaboratively with both onsite and offshore teams. Technical Documentation: Develop and review technical documentation for delivered artifacts. Problem Solving & Triage: Solve complex data-driven scenarios and triage defects and production issues. Deployment & Release: Participate in code release and production deployment. Continuous Learning: Demonstrate an ability to learn-unlearn-relearn concepts with an open and analytical mindset, and be comfortable tackling new challenges and ways of working. Mandatory Skills & Experience Technical Proficiency: PySpark Expertise: Extensive experience in design, build, and deployment of PySpark-based applications (minimum 3 years). Hadoop Ecosystem: Minimum 3 years of experience in HIVE, YARN, HDFS . Spark: Ability to design, build, and unit test applications on the Spark framework on Python . Scripting & Databases: Proficiency in Unix shell scripting and experience with RDBMS . Strong hands-on experience writing complex SQL queries , exporting, and importing large amounts of data using utilities. NoSQL Databases: In-depth knowledge of various NoSQL databases . CI/CD Tools: Experience in creating and maintaining integration and regression testing frameworks on Jenkins integrated with Bitbucket and/or GIT repositories . Code Quality: Ability to build abstracted, modularized, and reusable code components. Big Data Environment: Expertise in handling complex large-scale Big Data environments (preferably 20TB+). Experience & Qualifications: Minimum 3 years of extensive experience in design, build, and deployment of PySpark-based applications. BE/B.Tech/ B.Sc. in Computer Science/ Statistics from an accredited college or university (Preferred). Prior experience with ETL tools, preferably Informatica PowerCenter , is advantageous. Essential Professional Skills Problem Solving: Ability to solve complex data-driven scenarios and triage defects and production issues. Adaptability & Learning: Able to quickly adapt and learn, comfortable tackling new challenges and new ways of working, and ready to move from traditional methods to agile ones. Communication & Collaboration: Excellent communication skills, strong collaboration and coordination across various teams, and comfortable challenging peers and leadership. Customer Centricity: Good Customer Centricity, strong Target & High Solution Orientation. Initiative: Able to jump into an ambiguous situation and take the lead on resolution. Agile Mindset: Comfortable moving from traditional methods and adapting to agile ones. Proactiveness: Can prove self quickly and decisively.
