Data Engineer

Job Description

Data Engineer Job Description Key skills and Technical Competencies Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience 6+ years of experience at various levels of Data Engineering roles including 3 years in technical lead role managing end to end solution. Hands on Experience with Azure Databricks using PYSPARK with ability to do cluster capacity planning and workload optimization. Experience in designing solutions using databases and data storage technology using RDBMS (MS SQL Server) Experience building and optimizing Big Data data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models implemented through custom Experience in Azure Data factory. Proficiency in Python scripting. Experienced in using different python modules used for data munging Be up to date with data processing technology / platforms such as Spark (Databricks) Experience in Data Modeling for optimizing the solution performance Experienced in Azure DevOps with exposure to CI/CD. Good understanding of infrastructure components and their fit in different types of data solutions Experience of designing solutions deployed on Microsoft and Linux operating systems Experience of working in an agile environment, within a self-organising team.