Job
Description
Responsibilities: -Design and build scalable data pipelines using Azure Data Factory, Synapse Pipelines, and Spark (via Synapse or Databricks). -Implement complex data integration efforts across a wide range of structured, semi-structured, and unstructured sources (e.g., SQL, APIs, Event Hubs, Blob Storage, SharePoint). -Develop and optimize advanced transformations including joins, aggregations, window functions, slowly changing dimensions (SCDs), and schema evolution handling. -Establish and automate data quality checks, validations, and alerting for data anomalies using tools like Azure Data Quality, custom Python scripts. -Implement robust data lineage tracking, auditing, and error handling mechanisms. -Enforce data governance and compliance policies using Microsoft Purview, RBAC,Managed Identities, and Key Vault integration and data masking techniques. -Support data modeling activities (e.g., star/snowflake schemas, lakehouse structures) for reporting and analytics in Synapse and Power BI. -Collaborate with BI, ML, and application teams for end-to-end data integration and delivery. -Lead code reviews, mentor junior engineers, and ensure adherence to best practices and documentation. -Integrate DevOps pipelines for CI/CD deployment of data infrastructure using Azure DevOps, ARM/Bicep, or Terraform. Qualifications: -8+ years of experience in Data Engineering roles, with 2+ in senior or lead roles. -Strong expertise in Azure Data Factory, Synapse Analytics, Azure Data Lake, Delta Lake, and SQL. -Proficient in Python, Spark, and data orchestration. -Deep experience with data integration, API data ingestion, and real-time/streaming ingestion using Event Hubs, Kafka, or similar tools. -Proven track record implementing data quality frameworks and data validation logic. -Experience deploying data solutions in Fabric Workspaces and managing lifecycle through CI/CD tools (e.g., Azure DevOps, GitHub Actions) -Solid understanding of data warehousing, data lakehouse architecture, and data governance -Strong problem-solving skills and ability to drive architecture decisions collaboratively.
