Manager Data Engineering

Your role at Prudential is to design, build, and maintain data pipelines to ingest data from multiple sources into the cloud data platform. It is essential to ensure that these pipelines are constructed according to defined standards and documented comprehensively. Data governance standards must be adhered to and enforced to maintain data integrity and compliance. Additionally, you will be responsible for implementing data quality rules to ensure the accuracy and reliability of the data. As part of your responsibilities, you will need to implement data security and protection controls around Databricks Unity Catalog. You will be utilizing Azure Data Factory, Azure Databricks, and other Azure services to build and optimize data pipelines. Proficiency in SQL, Python/PySpark, and other programming languages for data processing and transformation is crucial. Staying updated with the latest Azure technologies and best practices is essential for this role. You will also provide technical guidance and support to team members and stakeholders. Detailed documentation of data pipelines, processes, and data quality rules must be maintained. Debugging, fine-tuning, and optimizing large-scale data processing jobs will be part of your routine tasks. Generating reports and dashboards to monitor data pipeline performance and data quality metrics is also important. Collaboration with data teams across Asia and Africa to understand data requirements and deliver solutions will be required in this role. Overall, your role at Prudential will involve designing, building, and maintaining data pipelines, ensuring data integrity, implementing data quality rules, and collaborating with various teams to deliver effective data solutions.,