Databricks Platform Administrator

Roles and Responsibilities

A Databricks Platform Administrator is a crucial role responsible for the effective design, implementation, maintenance, and optimization of the Databricks Lakehouse Platform within an organization. This individual ensures the platform is scalable, performant, secure, and aligned with business objectives, providing essential support to data engineers, data scientists, and analysts.

Key Responsibilities:

1. Provision and configure Databricks workspaces, clusters, pools, and jobs across environments.

2. Create catalogs, schemas, access controls, and lineage configurations.

3. Implement identity and access management using account groups, workspace-level permissions, and data-level governance.

4. Monitor platform health, cluster utilization, job performance, and cost using Databricks admin tools and observability dashboards.

5. Automate workspace onboarding, schema creation, user/group assignments, and external location setup using Terraform, APIs, or CLI.

6. Integrate with Azure services like ADLS Gen2, Azure Key Vault, Azure Data Factory, and Azure Synapse.

7. Support model serving, feature store, and MLflow lifecycle management for Data Science/ML teams.

8. Manage secrets, tokens, and credentials securely using Databricks Secrets and integration with Azure Key Vault.

9. Define and enforce tagging policies, data masking, and row-level access control using Unity Catalog and Attribute-Based Access Control (ABAC).

10. Ensure compliance with enterprise policies, security standards, and audit requirements.

11. Coordinate with Ops Architect, Cloud DevOps teams for network, authentication (e.g., SSO), and VNET setup.

12. Troubleshoot workspace, job, cluster, or permission issues for end users and data teams.

Preferred Qualifications:

• Databricks Certified Associate Platform Administrator or other relevant Databricks certifications.
• Experience with Apache Spark and data engineering concepts.
• Knowledge of monitoring tools (e.g., Splunk, Grafana, Cloud-native monitoring).
• Familiarity with data warehousing and data lake concepts.
• Experience with other big data technologies (e.g., Hadoop, Kafka).
• Previous experience leading or mentoring junior administrators.