Data Engineer - Big Data

As a Data Engineer, you will play a crucial role in designing and developing business-critical backend systems using stream processors and high-quality data pipelines. Your responsibilities will include assembling large, complex data sets to meet both functional and non-functional business requirements. You will be tasked with building the necessary infrastructure for optimal extraction, transformation, and loading of data from various sources using SQL and Azure "big data" technologies. Additionally, you will be responsible for constructing a cloud-native, real-time stream processing and data lake platform that can scale into the Zettabytes and beyond. Developing a data pipeline to offer actionable insights into customer acquisition, operational efficiency, and other vital business performance metrics will also be a key part of your role. Automation of data pipelines using DevOps tools will be expected from you. In this position, you will need to perform root cause analysis on both external and internal processes and data to identify opportunities for improvement. Building processes that support data transformation, workload management, data structures, dependency, and metadata will be essential. Being adaptable to change and always open to adopting better technologies will be crucial in this role. Collaborating with stakeholders to design and code large-scale batch and real-time data pipelines on Azure will be a part of your daily routine. You will also be expected to conduct code reviews and lead by example on code refactoring for readability, extensibility, and testability. Requirements: - A bachelor's degree in computer science or equivalent - Minimum of 4 years of experience as a Data Engineer - Proficiency in Azure big data tools such as Azure Databricks, Azure Synapse, HDInsight, and ADLS - Experience with relational SQL and NoSQL databases - Strong problem-solving and analytical skills in handling structured and unstructured datasets using Azure big data tools - Familiarity with data pipeline and workflow management tools like ADF and Logic Apps - Hands-on experience with Azure cloud services including VM, Azure Databricks, Azure SQL DB, and Azure Synapse - Knowledge of stream-processing systems such as Azure Streams - Proficiency in scripting languages like Python and strong expertise in developing Python notebooks. This role will offer you the opportunity to work on cutting-edge technologies and make a significant impact on the business by enabling data-driven decisions and insights.,