Job
Description
Job Description Location: Bengaluru, KA Position Type: Full-Time Hours: 9:00am to 5:00pm Weekdays (Monday – Friday) Key Responsibilities User & Access Management: Creating users, setting roles, managing access controls. Cluster & Resource Management: Provisioning clusters, monitoring resource usage, optimizing performance. Job Management: Scheduling, managing, and optimizing jobs and workflows. Monitoring & Troubleshooting: Monitoring system performance, troubleshooting issues. Platform Maintenance: Managing updates, backups, integrations, and disaster recovery. Collaboration: Working with data teams and cloud administrators to optimize and maintain the environment. Cost Optimization: Monitoring resource usage and controlling costs. Technical Experience Cloud platform expertise (AWS, Azure, GCP) Familiarity with the major cloud providers (AWS, Azure, Google Cloud). Databricks-specific admin experience (clusters, jobs, workspaces) Managing Databricks workspaces, clusters, jobs, and libraries. Configuring and scaling Databricks clusters, including creating cluster policies. Understanding Databricks security settings, including access control and user management. Knowledge of data security and governance, especially using Databricks’ security features such as secret management and data encryption. Security and compliance management Understanding of role-based access control (RBAC) and fine-grained access control (FGAC) in Databricks. Knowledge of security auditing and the ability to monitor logs for suspicious activities. Experience with Apache Spark and data engineering. Experience with Apache Spark and distributed computing (Databricks is built on top of Spark). Familiarity with Delta Lake (Databricks’ optimized data lake). Understanding data pipelines and how to schedule, monitor, and manage them. Automation and scripting skills (Python, Scala, SQL) Automation : Experience with automating workflows in Databricks, such as creating and managing clusters, jobs, and notebooks. Scripting languages : Strong skills in Python , Scala , or SQL for managing tasks or interacting with the Databricks API. Familiarity with REST APIs to integrate Databricks with other systems or automate tasks. Monitoring, troubleshooting, and performance optimization Familiarity with Databricks monitoring tools (e.g., cluster metrics, job logs). Ability to monitor resource usage and performance, troubleshoot issues, and optimize resource allocation. Knowledge of debugging Spark jobs and resolving common issues related to cluster failures, performance bottlenecks, and resource contention. Collaboration and communication Ability to collaborate with data scientists, data engineers, and other teams to ensure the Databricks environment meets their needs. Communication skills to effectively explain technical concepts to non-technical stakeholders. Problem-Solving: Ability to troubleshoot issues and resolve them efficiently. Adaptability: Keeping up with evolving technologies and best practices.
