Sr. Engineer (MS Fabric Migration & Development)

Role Overview: You will be responsible for hands-on migration and development tasks, including pipeline translation, stored procedure conversion, data modeling, large data handling, unit & integration testing, and performance tuning. Additionally, you will collaborate with the Technical Lead to optimize Spark code and Warehouse queries. Key Responsibilities: - Directly develop and migrate 20+ complex data pipelines from Azure Data Factory (ADF) into optimized Fabric Data Pipelines and Spark Notebooks (PySpark/Scala). - Rewrite and optimize 30+ complex stored procedures into robust, scalable transformations using PySpark in Fabric Lakehouses or high-performance T-SQL in Fabric Data Warehouses. - Implement the designed data architecture (Medallion layers: Bronze, Silver, Gold) within OneLake, focusing on Delta Lake best practices and V-Order optimization for efficient querying. - Execute the migration of 30+ TB of historical data efficiently and accurately, ensuring data integrity and reconciliation between legacy and Fabric environments. - Develop and execute comprehensive unit and integration tests to validate the correctness and performance of migrated data pipelines and stored procedures. - Work closely with the Technical Lead to identify and resolve data bottlenecks and optimize Spark code and Warehouse queries to meet latency requirements. - Operate within the established DevOps pipeline, ensuring all code changes, notebooks, and object definitions are managed under Git version control and deployed via the automated CI/CD framework. Qualifications Required: - 5+ years of dedicated experience in ETL/ELT development and data warehousing, with involvement in at least one major cloud data migration. - Strong, demonstrable experience with the Azure Data Stack (ADF, Synapse) and hands-on development experience in Microsoft Fabric. - Highly proficient in T-SQL for querying and procedural logic, and mandatory expertise in Python (PySpark) for data transformation. - Solid understanding of Kimball-style data warehousing, dimensional modeling, and modern Lakehouse concepts (Delta Lake, Parquet). - Experience working with Git for source control and familiarity with the principles of CI/CD deployment.,