Job
Description
We are seeking a highly skilled Senior Azure Databricks Data Engineer to design, develop, and optimize data solutions on Azure . The ideal candidate will have expertise in Azure Data Factory (ADF), Databricks, SQL, Python , and experience working with SAP IS-Auto as a data source . This role involves data modeling, systematic layer modeling, and ETL/ELT pipeline development to enable efficient data processing and analytics. Experience: 6+ years Key Responsibilities: Develop & Optimize ETL Pipelines : Build robust and scalable data pipelines using ADF, Databricks, and Python for data ingestion, transformation, and loading. Data Modeling & Systematic Layer Modeling : Design logical, physical, and systematic data models for structured and unstructured data. Integrate SAP IS-Auto : Extract, transform, and load data from SAP IS-Auto into Azure-based data platforms. Database Management : Develop and optimize SQL queries, stored procedures, and indexing strategies to enhance performance. Big Data Processing : Work with Azure Databricks for distributed computing, Spark for large-scale processing, and Delta Lake for optimized storage . Data Quality & Governance : Implement data validation, lineage tracking, and security measures for high-quality, compliant data. Collaboration : Work closely with business analysts, data scientists, and DevOps teams to ensure data availability and usability. Required Skills: Azure Cloud Expertise : Strong experience in Azure Data Factory (ADF), Databricks, and Azure Synapse . Programming : Proficiency in Python for data processing, automation, and scripting. SQL & Database Skills : Advanced knowledge of SQL, T-SQL, or PL/SQL for data manipulation. SAP IS-Auto Data Handling : Experience integrating SAP IS-Auto as a data source into data pipelines. Data Modeling : Hands-on experience in dimensional modeling, systematic layer modeling, and entity-relationship modeling . Big Data Frameworks : Strong understanding of Apache Spark, Delta Lake, and distributed computing . Performance Optimization : Expertise in query optimization, indexing, and performance tuning . Data Governance & Security : Knowledge of RBAC, encryption, and data privacy standards . Preferred Qualifications: Experience with CI/CD for data pipelines using Azure DevOps. Knowledge of Kafka/Event Hub for real-time data processing. Experience with Power BI/Tableau for data visualization (not mandatory but a plus). Show more Show less
