Data Engineer

Job Description

Key Responsibilities: · Requirement Gathering: · Collaborate with stakeholders to gather and document data requirements. · Translate business needs into technical specifications for data solutions. · Data Pipeline Development: · Design, implement, and maintain data pipelines using Microsoft Fabric. · Create and manage ETL processes, ensuring efficient data flow. · SSIS Integration: · Develop and maintain ETL processes using SQL Server Integration Services (SSIS). · Optimize SSIS packages for performance and reliability. · CDC Pipelines: · Implement Change Data Capture (CDC) pipelines to track and manage data changes. · Ensure timely and accurate data updates across systems. · PySpark Proficiency: · Utilize PySpark for data transformation and processing tasks. · Develop data processing scripts to handle large datasets efficiently. · Collaboration: · Work closely with data analysts, data scientists, and other teams to ensure data integrity. · Participate in cross-functional projects to enhance data accessibility and usability. Required Skills: · Technical Proficiency: · Strong experience with Azure Data Factory, Azure Synapse, and Microsoft Fabric. · Proficient in PySpark for data processing and transformation. · Solid understanding of SQL Server and experience in writing complex queries. · Data Modeling: · Knowledge of data modeling concepts and best practices. · Experience in designing data architectures that support business needs. · Cloud Services: · Familiarity with cloud computing concepts and services, particularly within the Azure ecosystem. · Understanding of data storage solutions such as Azure Data Lake and Blob Storage. · Problem-Solving: · Strong analytical skills to troubleshoot data-related issues. · Ability to optimize data workflows for performance and efficiency. Show more Show less