Data Engineer

Role & responsibilities

Technologies: Azure Data Factory, Data Bricks, SQL, Azure DevOps, Azure Azure Devops, gitlab, Power BI

Key Responsibilities

• Design and implement Extract, Transform, Load (ETL) processes to move data from various sources (on-premises, cloud, and third-party APIs) to Azure data platforms.
• Integrate data from diverse sources into Azure-based systems like Azure Data Lake/Azure SQL Database.
• Use Azure Data Factory or Data Bricks orchestration tools to automate and schedule data pipelines/Job workflows
• Design efficient data models (e.g., star or snowflake schema) for use in analytical applications and reporting systems.
• Optimize SQL queries and scripts for performance, ensuring low-latency and efficient data processing
• Continuously monitor the health of pipelines, jobs and infrastructure, ensuring they are running efficiently and securely
• Assist in building dashboards and reporting solutions using tools like Power BI, ensuring data is made available in an user-friendly format
• Address and resolve data issues, including failures in ETL pipelines, system performance problems, and data inconsistencies
• Ensure that all data engineering processes comply with industry security standards and best practices, including encryption, access controls, and data masking
• Keep logs of pipeline performance, failures, and system updates to provide visibility for monitoring and troubleshooting

Preferred candidate profile

Qualifications

• Bachelor's degree in B.E in Computer Science, Data Management, or a related field.
• Proficiency in Azure, Data Bricks, Azure Data Factory, SQL, Azure DevOps, gitlab and data visualization tools like Power BI.
• Strong understanding of data management and automation.
• Excellent analytical, troubleshooting and problem-solving skills.
• Years of Experience 6-7 years in a similar role.