Data Engineer

Job Title: Data Engineer Azure Fabric, PySpark & Migration Specialist

Position Summary

Key Responsibilities

• Lead the migration of legacy data pipelines and stored procedures from Azure Synapse and SQL Server into Microsoft Fabric using PySpark and Fabric-native capabilities.

• Design and implement PySpark-based transformations within Fabric pipelines to support scalable, efficient data processing.

• Develop, schedule, and monitor workflows using Apache Airflow, and support the transition to integrated orchestration in Fabric where applicable.

• Refactor and optimize existing ADF (Azure Data Factory) pipelines, integrating them into Fabric or coordinating them with Airflow as needed.

• Implement and maintain robust dimensional data models and support data marts optimized for analytics and reporting.

• Leverage Delta Lake format for reliable, performant data storage and support transactional workloads in OneLake.

• Ensure data quality, lineage, and governance through best practices in data engineering and observability.

• Collaborate closely with architects, analysts, and business stakeholders to deliver trusted, timely, and high-quality data solutions.

• Participate in DevOps processes including version control, automated testing, and CI/CD for data workflows.

Required Qualifications

• 4+ years of experience in data engineering with a strong focus on PySpark and distributed data processing.

• Proven experience with Apache Airflow for data pipeline orchestration.

• Solid hands-on experience in Azure Fabric, ADF, Azure Synapse, and SQL Server.

• Deep proficiency in PySpark, including performance tuning and optimization in large-scale data environments.

• Expertise in working with Delta Lake format and cloud-based data lakes, especially OneLake.

• Strong skills in T-SQL, KQL, and cloud-based data modeling (e.g., star schema, dimensional modeling).

• Understanding of data governance, security, and compliance best practices in Azure.

• Experience with version control (Git), automated deployments, and infrastructure as code.

• Familiarity with Power BI integration and semantic models in Fabric.

Preferred Qualifications

• Experience modernizing ETL/ELT workloads into Microsoft Fabric and lakehouse architectures.

• Exposure to real-time streaming data or hybrid batch-stream processing.

• Microsoft Azure certifications (e.g., Azure Data Engineer Associate) are a plus.

• Power BI Expertise

• Deep hands-on experience with Power BI Desktop and Service.

• Strong knowledge of Power BI data models (both Import and DirectQuery, DirectLake).

• Proficiency in DAX and MDX query writing and optimization.

• Experience with Row-Level Security, composite models, and performance tuning.

• Advanced MDX and SSAS Knowledge

• Experience writing and optimizing MDX queries for performance and readability.

• Understanding of multidimensional cubes and their structure.

• Comfortable with SSAS Multidimensional and SSAS Tabular models.

• Tools and Add-Ons

• Tabular Editor: Scripting, managing measures, KPIs, calculation groups, best practice checks.

• DAX Studio / MDX Studio: For query optimization, tracing queries, analyzing performance.

• ALM Toolkit: For model deployment and comparison/version control.

• Familiarity with tools like VertiPaq Analyzer, SQL Profiler for debugging and optimizing Power BI reports.