Senior Azure Data Engineer

Job Description

Responsibilities: Design andimplement data ingestion, transformation, and movement using Azure Data Factory(ADF), Azure Synapse, and Data Lake Collaborate withbusiness stakeholders, data scientists, and engineers to build robust datasolutions. Develop and manage high-volume, real-time, andbatch data ingestion pipelines using Azure Data Factory (ADF). Implementevent-driven architectures for real-time data movement and processing. Developlarge-scale data processing solutions using Azure Databricks and Apache Sparkwith PySpark, Scala, or Python. Optimize datapartitioning, caching, and indexing for efficient performance. Manage complextransformations and aggregations for structured and unstructured datasets. Design andimplement high-performance data models in Azure Synapse Analytics usingdedicated SQL Pools and Spark Pools. Optimize queryperformance, workload management, and cost efficiency in Synapse Analytics. Implement columnstore indexes, partitioning strategies, and data caching to enhanceperformance. Design andmanage secure, scalable data lakes using Azure Data Lake Storage Gen2 (ADLSGen2). Implement RBAC(Role-Based Access Control), encryption, and data masking to ensure security Implement Azure Monitor, Log Analytics, andApplication Insights for data pipeline monitoring and troubleshooting. Optimize cost management, auto-scaling, andperformance tuning across Azure services. Requirement: 8+ years of experience in data engineering, ETL development, and cloud-based data integration Strong experience with Azure Data Factory (ADF): ETL/ELT pipeline orchestration and data movement. Azure Databricks: Large-scale data transformation and big data processing with Apache Spark, PySpark, Scala, or Python. Azure Synapse Analytics: Data warehousing, SQL Pools, Spark Pools, and performance tuning. Azure Data Lake Storage Gen2 (ADLS Gen2): Secure, scalable data lake architecture. SQL Server: Advanced T-SQL, stored procedures, indexing, and query optimization.