Data Platform Engineer

Job Description

Project Role : Data Platform Engineer Project Role Description : Assists with the data platform blueprint and design, encompassing the relevant data platform components. Collaborates with the Integration Architects and Data Architects to ensure cohesive integration between systems and data models. Must have skills : Microsoft Azure Modern Data Platform Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Job Description: Development, customize and manage integration tools, databases, warehouses and analytical systems with the use of data related instruments/instances Create and run complex queries and automation scripts for operational data processing. Test the reliability and performance of each part of a system and cooperate with the testing team Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc. Responsible for setting up tools to view data, generate reports, and create visuals Monitoring the overall performance and stability of the system. Adjust and adapt the automated pipeline as data/models/requirements change. Mentor and train colleagues where necessary by helping them learn and improve their skills, as well as innovate and iterate on best practices. Solve complex issues with minimal supervision Make improvement and process recommendations that have an impact on the business Technology Stack Used & Required Knowledge: Mastery of Azure Data Services with minimum of 4 years of such as Azure Data Factory, Azure Synapse, Azure Databricks, Azure SQL Database, Azure Cosmos DB, Azure Blob Storage/Data Lake Storage, and Azure Stream Analytics. 5+ years Experience in using of Python/ PySpark for data engineering. Understanding of data types/ handling of different data models. Excellent understanding of ETL cycle Expertise in designing scalable and efficient data architectures on Azure is a plus Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plus. Good scripting and programming skills. Expertise in implementing data governance frameworks and ensuring compliance using Azure Purview. Familiarity with Azure Data Catalog for managing and discovering data assets. Understanding of Azure Key Vault for secure key management. Azure scripting with PowerShell/Azure CLI Others: Flexibility in using different technologies/ platforms Ability to shift gears quickly and cope with change Analytical & Detail Oriented Confident, proactive and self-motivated Disciplined and Team first approach oriented