Databricks Data Engineer

Key Responsibilities

• Develop & Optimize ETL Pipelines

• Build robust, scalable data pipelines using Azure Data Factory (ADF), Databricks, and Python.
• Handle data ingestion, transformation, and loading processes efficiently.

• Data Modeling & Systematic Layer Modeling

• Design logical, physical, and systematic models for both structured and unstructured data.

• SAP IS-Auto Integration

• Extract, transform, and load data from SAP IS-Auto into Azure-based data platforms.

• Database Management

• Develop and optimize SQL queries, stored procedures, and indexing strategies to improve performance.

• Big Data Processing

• Utilize Azure Databricks, Apache Spark, and Delta Lake for distributed and large-scale data processing.

• Data Quality & Governance

• Implement data validation, lineage tracking, and security controls to ensure high data quality and compliance.

• Cross-Functional Collaboration

• Collaborate with business analysts, data scientists, and DevOps teams to ensure data availability and usability.

• Testing and Debugging

• Write unit tests and debug issues to ensure a robust and error-free data implementation.
• Conduct performance optimization and security audits.

Required Skills and Qualifications

• Azure Cloud Expertise

• Strong experience in Azure Data Factory (ADF), Databricks, and Azure Synapse.

• Programming Skills

• Proficiency in Python for data processing, automation, and scripting tasks.

• SQL & Database Proficiency

• Advanced knowledge in SQL, T-SQL, or PL/SQL for efficient data manipulation.

• SAP IS-Auto Integration

• Proven experience in integrating SAP IS-Auto into data pipelines.

• Data Modeling Expertise

• Hands-on experience with dimensional modeling, systematic layer modeling, and ER modeling.

• Big Data Frameworks

• Strong knowledge of Apache Spark, Delta Lake, and distributed computing principles.

• Performance Optimization

• Expertise in query optimization, indexing, and performance tuning strategies.

• Data Governance & Security

• Understanding of RBAC, encryption methods, and data privacy standards.

Preferred Qualifications

• Experience with CI/CD pipeline setup using Azure DevOps.
• Familiarity with Kafka or Event Hub for real-time data streaming.
• Knowledge of Power BI or Tableau for data visualization (nice to have, not mandatory).