Job
Description
Position Purpose The Senior Developer will be a part of the ISPL Mumbai IHC ETL projects team. The developer position will primarily work on Apache Spark(python), Spark SQL, ETL tools, Unix, Autosys and DB Responsibilities Direct Responsibilities Expertise on PySpark, database migration, transformation, and integration solutions for any Data warehousing project. Must have excellent knowledge in Apache Spark and Python programming experience. Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations. Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred. Understanding of Unix/Linux + Shell Scripting Data modeling experience using advanced statistical analysis, unstructured data processing. Hands-on project experience on Jupyter Notebook/ Zeppelin/ PyCharm etc. IDEs Hands-on experience with AWS S3 Filesystem operations Contributing Responsibilities Good knowledge of Hadoop, Hive, and Cloudera/ Hortonworks Data Platform Strong hands-on experience in Processing Framework - Spark 2.x/3.x (Core, Spark SQL, Streaming) Language & Package - Python (Scripting & PySpark), Unix Shell, SQL Query (basic & advanced) Expertise in RDBM solutions (Postgres & Oracle) and NoSQL Databases Knowledge on Streaming Platform Apache Kafka, Spark Streaming Extensive hands-on experience in designing, building, and executing data pipeline using ETL/ELT tools. Big Data Hadoop - Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage HDFS (File Format Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning), Data Lake (Object Store) Optimize and troubleshoot existing PySpark applications for performance improvements. Technical & Behavioral Competencies Minimum 5 years hands-on experience with PySpark, Kubernetes, Docker Strong technical expertise in PySpark, Kubernetes, Docker and be in position to handle all technical difficulties Strong in designing (data warehousing) concepts. Good working knowledge in Unix/Ubuntu (Should be able to write wrapper scripts) Capable of tuning the code to handle the huge data volume. Responsible for translating/understanding the functional requirements to meet the specified technical requirements. Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied Involved in coding, testing, implementing, debugging, and documenting the complex programs. Involved in creating proper technical documentation in the work assignments. Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process. Ensures that all the standard requirements have been met and is involved in performing the technical analysis. Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements. Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming. Behavioral Competencies Excellent verbal and written communication skills. Conduct meetings with global stakeholders, prepare minutes & summaries. Assertiveness, Negotiation, Proactiveness & Prioritization skills are important. Discipline in documenting, following up on issues and changes. Experience in interacting with global stakeholders and independently managing discussions. Specific Qualifications (if required) Skills Referential Behavioural Skills : (Please select up to 4 skills) Communication skills - oral & written Ability to synthetize / simplify Attention to detail / rigor Organizational skills Transversal Skills: (Please select up to 5 skills) Analytical Ability Ability to understand, explain and support change Ability to develop and adapt a process Ability to manage / facilitate a meeting, seminar, committee, training Ability to develop and leverage networks Education Level: Bachelor Degree or equivalent Experience Level At least 5 years Other/Specific Qualifications (if required) Additional knowledge on reporting tools is advantage
