Job
Description
The Offshore Data Engineering Lead will be responsible for overseeing the data and application development efforts which support our Microsoft Data Mesh Platform. Working as a part of the Investec Central Data Team, the candidate will be responsible for leading development on solutions and applications that support our data domain teams with creation of data products. This role involves driving technical initiatives, exploring new technologies, and enhancing engineering practices within the data teams in-line with the group engineering strategy. The Data Engineering Lead will be a key driver for Investecs move to Microsoft Fabric and other enablement data quality, data management and data orchestration technologies. Key Roles and Responsibilities: Lead the development and implementation of data and custom application solutions that support the creation of data products across various data domains. Design, build, and maintain data pipelines using Microsoft Azure Data Platform, Microsoft Fabric and Databricks technologies. Ensure data quality, integrity, and security within the data mesh architecture. Share group engineering context with the CIO and engineers within the business unit continuously. Drive engineering efficiency and enable teams to deliver high-quality software quickly within the business unit Cultivate a culture focused on security, risk management, and best practices in engineering Actively engage with the data domain teams, business units and wider engineering community to promote knowledge sharing Spearhead technical projects and innovation within the business units engineering teams and contribute to group engineering initiatives Advance the technical skills of the engineering community and mentor engineers within the business unit Enhance the stability, performance, and security of the business units systems. Develop and promote exceptional engineering documentation and practices Build a culture of development and mentorship within the central data team Provide guidance on technology and engineering practices Actively encourages creating Investec open-source software where appropriate within the business unit Actively encourages team members within the business unit to speak at technical conferences based on the work being done Core Skills and Knowledge: Proven experience in data engineering, with a strong focus on Microsoft Data Platform technologies, including Azure Data Factory, Azure SQL Database, and Databricks Proficiency in programming languages such as C# and/or Python, with experience in application development being a plus Experience with CI/CD pipelines, Azure, and Azure DevOps Strong experience and knowledge with PySpark and SQL with the ability to create solutions using Microsoft Fabric Ability to create solutions that query and work with web APIs In-depth knowledge of Azure, containerisation, and Kubernetes Strong understanding of data architecture concepts, particularly data mesh principles Excellent problem-solving skills and the ability to work independently as a self-starter Strong communication and collaboration skills, with the ability to work effectively in a remote team environment Relevant degree in Computer Science, Data Engineering, or a related field is preferred
