Senior Data Engineer

We are searching for a Senior Data Engineer with significant experience in developing ETL processes utilizing PySpark Notebooks and Microsoft Fabric, as well as supporting existing legacy SQL Server environments. The perfect candidate will have a solid foundation in Spark-based development, showcase advanced SQL skills, and feel at ease working autonomously, collaboratively within a team, or guiding other developers when necessary, all while possessing excellent communication abilities. The ideal candidate will also demonstrate expertise with Azure Data Services, such as Azure Data Factory, Azure Synapse, or similar tools, familiarity with creating DAG's, implementing activities, and running Apache Airflow, and knowledge of DevOps practices, CI/CD pipelines, and Azure DevOps. Key Responsibilities: - Design, develop, and manage ETL Notebook orchestration pipelines utilizing PySpark and Microsoft Fabric. - Collaborate with data scientists, analysts, and stakeholders to grasp data requirements and provide effective data solutions. - Migrate and integrate data from legacy SQL Server environments into modern data platforms. - Optimize data pipelines and workflows for scalability, efficiency, and reliability. - Provide technical leadership and mentorship to junior developers and team members. - Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability. - Develop, maintain, and uphold data engineering best practices, coding standards, and documentation. - Conduct code reviews and offer constructive feedback to enhance team productivity and code quality. - Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms. Qualifications: - Bachelors or Masters degree in Computer Science, Data Science, Engineering, or a related field. - 10+ years of experience in data engineering, focusing on ETL development using PySpark or other Spark-based tools. - Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling. - Experience with Microsoft Fabric or similar cloud-based data integration platforms is advantageous. - Strong understanding of data warehousing concepts, ETL frameworks, and big data processing. - Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is a plus. - Experience dealing with both structured and unstructured data sources. - Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues. - Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools. - Experience of creating DAG's, implementing activities, and running Apache Airflow. - Familiarity with DevOps practices, CI/CD pipelines, and Azure DevOps. In conclusion, Aspire Systems is a global technology services firm that acts as a trusted technology partner for over 275 clients worldwide. Aspire collaborates with leading enterprises in Banking, Insurance, Retail, and ISVs to help them leverage technology for business transformation in the current digital era. The company's dedication to Attention. Always. reflects its commitment to providing care and attention to both its customers and employees. With over 4900 employees globally and a CMMI Level 3 certification, Aspire Systems operates in North America, LATAM, Europe, Middle East, and Asia Pacific. Aspire Systems has been consistently recognized as one of the Top 100 Best Companies to Work For by the Great Place to Work Institute for the 12th consecutive time. For more information about Aspire Systems, please visit https://www.aspiresys.com/.,