Azure Databricks Engineer

As an Azure Databricks Engineer at Dhanam Cloud Computing Private Limited, you will play a crucial role in building and maintaining end-to-end data pipelines using ADF, Databricks, or Fabric. Your responsibilities will include ingesting and transforming structured & unstructured data into ADLS / Lakehouse, developing scalable ETL/ELT solutions using PySpark, Python, and SQL, implementing Delta Lake best practices, and optimizing Spark jobs. You will collaborate with Data Science teams to deploy and operationalize ML models, ensuring data governance, security, and quality with tools like Unity Catalog and RBAC. Furthermore, you will be involved in implementing CI/CD pipelines using Git & Azure DevOps and supporting BI and analytics teams, including Power BI enablement. Key Responsibilities: - Build & maintain end-to-end data pipelines using ADF, Databricks, or Fabric - Ingest and transform structured & unstructured data into ADLS / Lakehouse - Develop scalable ETL/ELT solutions using PySpark, Python, and SQL - Implement Delta Lake best practices and optimize Spark jobs - Work with Data Science teams to deploy and operationalize ML models - Ensure data governance, security, and quality (Unity Catalog, RBAC) - Implement CI/CD pipelines using Git & Azure DevOps - Support BI and analytics teams, including Power BI enablement Qualifications Required: - 3-5 years of experience in Azure-based Data Engineering - Strong hands-on experience with ADF, ADLS, Synapse, Databricks/Fabric - Proficiency in SQL, Python, PySpark, and Spark performance tuning - Good understanding of ETL, data modeling, warehousing, and Lakehouse architecture - Familiarity with Git, DevOps, and pipeline automation - Exposure to AI/ML workflows is a plus Please note that Azure / Databricks / Fabric certifications and experience with MLflow or advanced Databricks features are preferred but not mandatory for this role. As an Azure Databricks Engineer at Dhanam Cloud Computing Private Limited, you will play a crucial role in building and maintaining end-to-end data pipelines using ADF, Databricks, or Fabric. Your responsibilities will include ingesting and transforming structured & unstructured data into ADLS / Lakehouse, developing scalable ETL/ELT solutions using PySpark, Python, and SQL, implementing Delta Lake best practices, and optimizing Spark jobs. You will collaborate with Data Science teams to deploy and operationalize ML models, ensuring data governance, security, and quality with tools like Unity Catalog and RBAC. Furthermore, you will be involved in implementing CI/CD pipelines using Git & Azure DevOps and supporting BI and analytics teams, including Power BI enablement. Key Responsibilities: - Build & maintain end-to-end data pipelines using ADF, Databricks, or Fabric - Ingest and transform structured & unstructured data into ADLS / Lakehouse - Develop scalable ETL/ELT solutions using PySpark, Python, and SQL - Implement Delta Lake best practices and optimize Spark jobs - Work with Data Science teams to deploy and operationalize ML models - Ensure data governance, security, and quality (Unity Catalog, RBAC) - Implement CI/CD pipelines using Git & Azure DevOps - Support BI and analytics teams, including Power BI enablement Qualifications Required: - 3-5 years of experience in Azure-based Data Engineering - Strong hands-on experience with ADF, ADLS, Synapse, Databricks/Fabric - Proficiency in SQL, Python, PySpark, and Spark performance tuning - Good understanding of ETL, data modeling, warehousing, and Lakehouse architecture - Familiarity with Git, DevOps, and pipeline automation - Exposure to AI/ML workflows is a plus Please note that Azure / Databricks / Fabric certifications and experience with MLflow or advanced Databricks features are preferred but not mandatory for this role.