Azure Databricks Engineer

Key Skills:

Roles and Responsibilities:

• Design and build scalable data pipelines using Azure Databricks and Delta Lake.
• Perform data transformation, cleansing, and enrichment using PySpark and Python.
• Implement and optimize Big Data solutions in Azure cloud environments.
• Develop and manage ETL workflows with high performance and reliability.
• Write optimized SQL queries for analytics and reporting over large datasets.
• Leverage Azure DevOps for CI/CD, story tracking, and deployment automation.
• Integrate data sources and services across the Azure ecosystem.
• Conduct data analysis and troubleshooting to ensure accuracy and consistency.
• Collaborate with cross-functional teams in Agile/Scrum projects to deliver sprint goals.

Skills Required:

• Strong proficiency in Azure Databricks for building and managing data pipelines.
• Hands-on experience with PySpark and Python for data transformation and analytics.
• Expertise in Azure ecosystem (Data Lake, Data Factory, Synapse, Key Vault, etc.).
• Solid understanding of Big Data concepts and data lake architecture.
• Experience in Delta Lake, CI/CD pipelines using Azure DevOps, and version control (Git).
• Proficiency in SQL for complex querying and data manipulation.
• Knowledge of data integration, ETL processes, and data governance principles.
• Familiarity with Agile methodologies and sprint-based delivery.
• Strong analytical, debugging, and communication skills.

Education: