Senior Azure Data Engineer

As a Senior Azure Developer at our company, you will be a key player in designing, developing, and implementing robust and scalable data solutions on the Microsoft Azure platform. Your responsibilities will include: - Designing and implementing highly performant data ingestion pipelines from various sources using Azure Databricks, Azure Data Factory, and other Azure services. - Translating complex business requirements into technical specifications and robust data solutions. - Contributing to the architectural design of data platforms and solutions on Azure. You will also be involved in: - Developing, testing, and deploying data transformation and processing jobs primarily using PySpark and Python within Databricks and Azure Synapse environments. - Creating and managing data pipelines using Azure Data Factory for orchestration, data movement, and transformation. - Developing scalable frameworks for ingesting and processing various data sets, including complex geospatial data. - Implementing data quality checks, validation, and monitoring mechanisms within data pipelines. Additionally, you will: - Optimize existing data pipelines and queries for performance, cost-efficiency, and scalability. - Troubleshoot and resolve complex data-related issues to ensure data accuracy and pipeline reliability. You will collaborate closely with data architects, data scientists, business analysts, and other engineering teams to deliver end-to-end data solutions. Your role may also involve mentoring junior developers and sharing knowledge within the team. In terms of qualifications, you should have: - 7 to 12 years of overall experience in data engineering or software development, with a significant focus on cloud-based data solutions. - Strong hands-on experience with Microsoft Azure data services. - In-depth experience with Azure Databricks, expert-level proficiency in PySpark, and strong programming skills in Python for data engineering tasks. - Extensive experience with Azure Data Factory for building and orchestrating complex ETL/ELT pipelines. - Solid understanding and practical experience with Azure Synapse Analytics and good knowledge of relational and non-relational database concepts. Nice to have skills include experience with Snowflake, knowledge of other Azure data services, experience with real-time data processing technologies, familiarity with CI/CD pipelines for data solutions, and Azure certifications. Please note that the above description is a summary of the key details from the job description provided.,