Data Engineer

You are an experienced Data Engineer with expertise in Databricks, Python, PySpark, and Power BI, possessing a strong background in data integration within the modern Azure ecosystem. Your role will be pivotal in designing, developing, and implementing scalable data engineering solutions. Your responsibilities will include designing and implementing robust data solutions using Azure Data Factory, Databricks, and related tools. You will be building and maintaining scalable ETL/ELT pipelines, focusing on performance and reliability. Writing efficient and reusable code using Python and PySpark, you will perform data cleansing, transformation, and migration across various platforms. Your hands-on experience with Azure Data Factory (ADF) for at least 1.5 to 2 years will be critical. Additionally, you will develop and optimize SQL queries, stored procedures, and manage large data sets using SQL Server, T-SQL, PL/SQL, etc. Collaborating with cross-functional teams, you will understand business requirements and provide data-driven solutions. Direct engagement with clients and business stakeholders to gather requirements, recommend optimal solutions, and ensure successful delivery is expected. Working with Power BI for basic reporting and data visualization tasks, along with a strong knowledge of data warehousing concepts, modern data platforms, and cloud-based analytics, is essential. Adhering to coding standards and best practices, including thorough documentation and testing (unit, integration, performance), will be part of your routine. Supporting the operations, maintenance, and enhancement of existing data pipelines and architecture, you will estimate tasks and plan release cycles. Your technical skills should include proficiency in Python, PySpark, Azure Data Factory, Databricks, Azure ecosystem, SQL Server, T-SQL, PL/SQL, Power BI (PBI), data warehousing, ETL/ELT, data cleansing, data migration, version control, Agile methodologies, and good problem-solving abilities. Qualifications include experience with coding in Pysense within Databricks (added advantage), a solid understanding of cloud data architecture and analytics processes, and the ability to independently initiate and lead conversations with business stakeholders.,