Databricks Specialist

As a Databricks Specialist at Birlasoft, your role involves designing, implementing, and maintaining the data platform using Databricks on the Microsoft Azure cloud. You will collaborate with cross-functional teams to ensure the scalability, reliability, and performance of analytics and data solutions. **Key Responsibilities:** - Manage and administer the Databricks platform, including the Unity Catalog, to support data engineering and data science workflows. - Work with Unity Catalog to organize, discover, and govern data assets within the Databricks platform. - Maintain the data quality and consistency within the Unity Catalog. - Collaborate with data engineers and data scientists to understand their requirements and configure the Databricks platform accordingly. - Create and manage workspaces, clusters, and jobs in the Databricks environment. - Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability. - Monitor and manage platform costs, including optimizing resource utilization to control expenses. - Implement security measures and access controls to ensure data privacy and protection within the Databricks platform. - Configure and optimize Databricks clusters for efficient and scalable data processing and analytics. - Stay up-to-date with security best practices and compliance requirements. - Monitor the performance and health of the Databricks platform and troubleshoot any issues that arise. - Setup Databricks MLOps. - Collaborate with data architects and data stewards to ensure data quality and consistency within the Unity Catalog. - Build, configure, and maintain CICD pipelines using GitHub and Azure Devops for the deployment of applications and infrastructure on Databricks. **Qualification Required:** - Candidate working hours would be Chicago time zone (18:30 IST to 2:30 IST). - Bachelors degree in Computer Science, Engineering, or related field. - Proven experience as a Databricks administrator or similar role. - Strong expertise in Databricks platform and its components, including workspaces, clusters, and jobs. - Experience in configuring and optimizing Databricks clusters for big data processing. - Familiarity with security measures and access controls within the Databricks platform. - Understanding of data governance principles and experience with data cataloging tools like Unity Catalog. - Experience with Infrastructure as Code (IaC) and automation tools, preferably Terraform. - Knowledge of data catalog tools (e.g., Microsoft Purview). - Excellent problem-solving skills and ability to work in a collaborative environment. - Relevant certifications in Databricks or related technologies are a plus.,