Azure Data Engineer

About the Role

We are seeking a Databricks Data Engineer with hands-on experience in designing, building, and optimizing data pipelines and lakehouse architectures. The role involves working on Databricks-based data migration projects, ensuring efficient data ingestion, transformation, and delivery across diverse platforms.

Key Responsibilities

Data Engineering & Implementation

• Design and build scalable data pipelines on Databricks using Spark, SQL, and Python/Scala.

• Develop and optimize data ingestion workflows using tools like ADF, DMS, or equivalent.

• Implement medallion architecture (bronze, silver, gold) for efficient data organization and lineage tracking.

Databricks Development & Configuration

• Work on core Databricks features like Delta Lake, Unity Catalog, and Serverless SQL.

• Manage ETL/ELT processes for structured and unstructured data.

• Ensure compliance with governance, metadata management, and performance best practices.

Collaboration & Documentation

• Collaborate with architects, analysts, and business stakeholders to understand data needs and deliver reliable solutions.

• Maintain technical documentation for data pipelines and workflows.

Quality & Optimization

• Conduct unit testing, performance tuning, and troubleshooting for data jobs.

• Continuously improve data quality, efficiency, and scalability.

Technical Skills Required

• 4-6 years of experience in data engineering, including Databricks development.

• Strong knowledge of Spark, SQL, Python/Scala, and ETL/ELT processes.

• Experience with ADF, AWS DMS, or other ingestion tools.

• Hands-on experience with Delta Lake, Unity Catalog, and Databricks SQL.

• Familiarity with Azure or AWS data ecosystem.

Soft Skills

• Excellent verbal and written communication in English.

• Strong analytical and problem-solving skills.

• Ability to work collaboratively in agile and cross-functional environments.