Software Engineer - Data Engineer

You are a Data Engineer with expertise in Databricks, PySpark, SQL, and Azure Data Factory (ADF) to design and optimize scalable data pipelines. You will be responsible for designing, developing, and maintaining ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks. Processing and transforming large datasets using PySpark in Databricks will also be a key aspect of your role. Additionally, you will write and optimize complex SQL queries for data extraction, transformation, and loading. Implementing data lakehouse architectures using Delta Lake in Databricks is part of the responsibilities, along with managing and optimizing Snowflake data warehouses and using DBT for modular and scalable data transformations. Ensuring data quality, governance, and monitoring across pipelines will be essential, and you will collaborate with data analysts and business teams to deliver actionable insights. The ideal candidate will have a proven track record in building efficient ETL/ELT processes, data warehousing, and cloud-based data solutions. You should have 4+ years of hands-on experience in Databricks, PySpark, SQL, and ADF. Strong expertise in Databricks, proficiency in Azure Data Factory (ADF), and advanced SQL skills are required. Experience with data warehousing concepts, Snowflake, and DBT is beneficial. Python scripting for automation and data processing is a bonus. Preferred qualifications include certifications in Databricks and Azure Data Engineering, experience with CI/CD pipelines, and knowledge of streaming data such as Kafka and Event Hubs. If you are looking to join a dynamic team where you can contribute your expertise and grow professionally, we encourage you to apply for this exciting opportunity. Further details about Beyond Key can be found at https://www.beyondkey.com/about.,