Data Engineer

Role Summary

We are seeking a highly skilled Data Engineer to design, develop, and maintain scalable, secure, and high-performance data solutions. The ideal candidate will be a strong analytical problem solver with the ability to independently design, develop, test, and deliver end-to-end data engineering solutions. The role requires hands-on experience with distributed data processing technologies, modern data platforms, and software engineering best practices, including testing and CI/CD.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion, transformation, and serving pipelines.
• Build and optimize data solutions using Azure Databricks, PySpark, Delta Lake, and Azure data services.
• Develop data integration workflows using Azure Data Factory and other Azure-native services.
• Design and implement data models that support analytics, reporting, and downstream applications.
• Implement robust data quality, validation, monitoring, and observability solutions.
• Develop automated testing strategies and maintain high code quality standards.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Collaborate with architects, product owners, analysts, and business stakeholders to deliver data products.
• Troubleshoot production issues and continuously improve platform reliability, scalability, and performance.
• Ensure adherence to data governance, security, and engineering best practices.

Mandatory Skills

• Programming skills in Python, Java, Scala, or another modern programming language.
• Advanced SQL skills.
• Hands-on experience with Apache Spark / PySpark.
• Experience with Databricks or similar distributed data processing platforms.
• Experience building scalable data pipelines and data integration solutions.
• Good understanding of data warehousing and dimensional modelling concepts.
• Proficiency with Git or any source control practices.
• Experience with automated testing frameworks such as PyTest, JUnit, or equivalent.
• Strong analytical, troubleshooting, and problem-solving skills.
• Knowledge of software engineering best practices, including code reviews, testing, and reusable design patterns.
• Experience developing and supporting production-grade data solutions.

Preferred / Good-to-Have

• Experience with Azure could.
• Experience with Databricks, Delta Lake, Delta Live Tables (DLT), or equivalent technologies.
• Experience with Azure Data Factory (ADF), Azure Synapse, AWS Glue, EMR, or comparable cloud-native data services.
• Experience with data quality, observability, and monitoring frameworks.
• Cloud or Databricks certifications.
• Experience working in Agile delivery environments.