Databricks Specialist

Job Description

Area(s) of responsibility About Birlasoft Birlasoft, a global leader at the forefront of Cloud, AI, and Digital technologies, seamlessly blends domain expertise with enterprise solutions. The company’s consultative and design-thinking approach empowers societies worldwide, enhancing the efficiency and productivity of businesses. As part of the multibillion-dollar diversified CKA Birla Group, Birlasoft with its 12,000+ professionals, is committed to continuing the Group’s 170-year heritage of building sustainable communities. Job Description As a Platform Admin, will play a crucial role in designing, implementing, and maintaining our data platform using Databricks on the Microsoft Azure cloud. You’ll collaborate with cross-functional teams to ensure the scalability, reliability, and performance of our analytics and data solutions. Title: Databricks Experience Level:Experience Level: Level 2 (Overall 5 to 7 yrs) Must Have Skills Must Have Skills: 2 to 4 years of exp in the below skill set Snowflake, Databricks, SQL, Python, Terraform, Azure Devops Good To Have Skills exp in Azure Cloud, Database services, DevOps Operation Job Qualification / Requirements Candidate working hours would be Chicago time zone (18:30 IST to 2:30 IST). Bachelor’s degree in Computer Science, Engineering, or related field. Proven experience as a Databricks administrator or similar role. Strong expertise in Databricks platform and its components, including workspaces, clusters, and jobs. Experience in configuring and optimizing Databricks clusters for big data processing. Familiarity with security measures and access controls within the Databricks platform. Understanding of data governance principles and experience with data cataloging tools like Unity Catalog. Experience with Infrastructure as Code (IaC) and automation tools, preferably Terraform. Knowledge of data catalog tools (e.g., Microsoft Purview). Excellent problem-solving skills and ability to work in a collaborative environment. Relevant certifications in Databricks or related technologies are a plus. Job Responsibilities Manage and administer the Databricks platform, including the Unity Catalog, to support data engineering and data science workflows. Work with Unity Catalog to organize, discover, and govern data assets within the Databricks platform. Maintain the data quality and consistency within the Unity Catalog. Collaborate with data engineers and data scientists to understand their requirements and configure the Databricks platform accordingly. Create and manage workspaces, clusters, and jobs in the Databricks environment. Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability. Monitor and manage platform costs, including optimizing resource utilization to control expenses. Implement security measures and access controls to ensure data privacy and protection within the Databricks platform. Configure and optimize Databricks clusters for efficient and scalable data processing and analytics. Stay up-to-date with security best practices and compliance requirements. Monitor the performance and health of the Databricks platform and troubleshoot any issues that arise. Monitor and manage platform costs, including optimizing resource utilization to control expenses. Setup Databricks MLOps Collaborate with data architects and data stewards to ensure data quality and consistency within the Unity Catalog. Build, Configure and maintain CICD pipelines using GitHub and Azure Devops for the deployment of application and infrastructure on Databricks.