Data Engineer Pyspark

The ideal candidate for this role should possess a Post Graduate Degree in Computer Science or a related field (or equivalent industry experience) with a strong background in Mathematics and Statistics, complemented by a minimum of 5+ years of development and design experience as a Data Engineer. Required expertise includes experience with Big Data platforms and distributed computing technologies such as Hadoop, Map/Reduce, Spark, HBase, and Hive. The role demands proficiency in data pipeline software engineering best practices in Python, encompassing linting, unit/integration tests, git flow/pull request process, object-oriented development, data validation, algorithms, data structures, technical troubleshooting, debugging, and bash scripting. Experience is also needed in Data Quality Assessment (profiling, anomaly detection), data documentation (schema, dictionaries), data architecture, data warehousing, and modelling techniques (Relational, ETL, OLTP), with the ability to consider performance alternatives. Candidates should be proficient in SQL, PL/SQL, or T-SQL within RDBMS production environments, with No-SQL databases being a plus, and have Linux OS configuration and shell scripting skills. A strong understanding and practical experience with Agile, DevOps, and CI/CD principles (GitHub, Jenkins, etc.) are essential, along with active involvement in solving and troubleshooting issues in a distributed services ecosystem. The engineer must ensure the quality of technical and application architecture and design, effectively research and benchmark technology, and preferably have experience in a Banking, Financial, or Fintech enterprise environment. They should be able to influence multiple teams on technical considerations, acting as a self-motivator and self-starter with the ability to drive things without supervision while collaborating across the organization. Finally, excellent soft and interpersonal skills are mandatory to interact with the team, stakeholders, and management, including good listening skills, clear speaking, a positive attitude, and the ability to establish a climate of trust and a motivating environment within the team.