Sr. Azure Data Engineer

Job Description

Roles and Responsibilities : Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Databricks. Collaborate with cross-functional teams to understand business requirements and design scalable solutions for big data processing using PySpark on Azure Databricks. Develop complex SQL queries to optimize database performance and troubleshoot issues in Azure SQL databases. Ensure high availability of critical systems by implementing monitoring tools such as Prometheus and Grafana. Job Requirements : Experience in designing and developing large-scale data pipelines using ADF or similar technologies. Strong proficiency in Python programming language with experience working with libraries like Pandas, NumPy, etc. Experience working with Azure Databricks platform including creating clusters, managing workloads, and optimizing resource utilization. Proficiency in writing complex SQL queries for querying relational databases.