Azure Databricks Engineer - Medallion Architecture

Description

Azure Databricks Engineer

Notice Period :

Primary Skills :

• At least 1 real-time project Exp is a must
• DBT Core / DBT Cloud
• Spark SQL, Python Programming Skills
• Databricks certifications (Associate or Professional) Preferred
• ADF (Azure Data Factory) & Azure DevOps Exp.- Preferred
  

• 5+ yrs. in Azure Databricks technologies.
• Experience in Medallion Architecture is mandatory.
• Good understanding of ETL , BI and DWH technologies.
• Hands-on experience on Jason or XML files as sources.
• Experience in autoloader for incremental loading and explode options for flattening data
• Experience in loading Delta live tables
• Experience in PySpark or Spark using data frames.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.
• Work together with data scientists and analysts to understand the needs for data and create effective data workflows.
• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Utilizing Azure Data Bricks to create and maintain ETL (Extract, Transform, Load) operations.
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
• Monitoring and resolving data pipeline problems to guarantee consistency and availability of the data.
• Good communication skills Oral & Written