Data Engineer

Key Skills

Roles and Responsibilities:

• Collaborate with technical architects and cloud solutions teams to architect and design data pipelines, data marts, and reporting solutions.
• Develop, test, and optimize Databricks jobs for efficient data processing, ETL tasks, and report generation.
• Leverage Databricks to set up scalable data pipelines that integrate with diverse data sources and cloud platforms.
• Ensure adherence to best practices for code quality, data security, and scalability.
• Participate in code and design reviews to maintain high standards of development.
• Optimize data querying layers to improve performance and support analytical requirements.
• Collaborate with data scientists and analysts to support machine learning workflows and analytical needs.
• Stay current with the latest developments in Databricks and associated technologies to drive innovation.

Experience Requirement:

• 3 to 8 years of relevant experience in data engineering.
• Proficiency in PySpark or Scala and SQL for data processing tasks.
• Hands-on experience with Azure Databricks, Delta Lake, Delta Live Tables, Auto Loader, and Databricks SQL.
• Strong expertise with Azure Data Lake Storage (ADLS Gen2).
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Experience using Azure Synapse and Power BI for analytics and reporting is preferred.
• Familiarity with Spark Streaming, MLOps, and Azure Resource Manager (ARM) templates is an added advantage.
• Proven experience building and optimizing data ingestion and transformation pipelines using Databricks, Azure Synapse, and Azure Data Factory.
• Practical experience integrating Databricks with Synapse, ADLS Gen2, and Power BI for end-to-end analytics solutions.
• Experience in developing, optimizing, and deploying Power BI reports is a plus.

Education: