Azure Databricks Lead Engineer

Alcor Solutions is a leading digital transformation and cloud consulting firm, enabling enterprises to leverage modern data platforms, cloud-native solutions, and emerging technologies to drive innovation and business outcomes.

We are seeking an Azure Databricks Lead Engineer to architect, administer, and scale our enterprise Databricks environment while establishing best practices and governance across the organization.

Role Overview

The Azure Databricks Lead Engineer will serve as the subject matter expert (SME) for our Databricks platform, overseeing its architecture, administration, and optimization across the enterprise.

This individual will be responsible for designing the Databricks Medallion architecture, implementing Unity Catalog governance, managing cluster policies, and driving the adoption of Databricks best practices. The role requires a combination of hands-on administration and strategic platform design, ensuring Databricks is secure, scalable, and fully aligned with enterprise data initiatives.

• Design and implement

  Databricks Medallion architecture

  for enterprise-wide data usage.
• Define

  workspace strategy

  , including design, configuration, and organization for multi-team and multi-project usage.
• Establish

  best practices

  for data ingestion, transformation, and consumption using Databricks.
• Administer and optimize the

  Azure Databricks platform

  (clusters, pools, compute policies).
• Set up and manage

  Unity Catalog

  , including access control, catalog/schema security, and governance for

  PII and PHI

  data.
• Create and enforce

  cluster management policies

  , workspace permissions, and

  environment segregation

  (dev/test/prod).
• Monitor

  platform health, usage, and costs

  ; implement scaling and tuning strategies.
• Collaborate with

  cloud infrastructure, networking, and security teams

  to ensure Databricks integrates with enterprise systems and policies.
• Support

  data engineers, data scientists, and analytics teams

  by enabling seamless deployment of applications and pipelines on Databricks.
• Act as the

  primary technical advisor and escalation point

  for Databricks-related issues.
• Define processes for

  sizing and provisioning Databricks clusters

  based on project requirements.
• Establish governance for

  non-production and production environments

  to ensure compliance and cost-efficiency.
• Build guidelines for

  version control, Unity Catalog repository configuration, and DevOps pipelines

  .
• Serve as the

  Databricks SME

  across the enterprise, evangelizing best practices and new features.
• Mentor internal teams and

  conduct workshops or training sessions

  to upskill stakeholders on Databricks usage.
• Stay current with

  Databricks and Azure advancements

  , ensuring the enterprise platform evolves with industry standards.

Experience & Qualifications:

• 7+ years

  of relevant experience in data engineering, cloud data platforms, or similar roles.
• Proven track record of

  Databricks administration and architecture

  , ideally with

  formal Medallion architecture experience

  .
• Experience in managing

  large-scale Databricks environments

  across multiple teams and projects.
• Deep expertise in

  Azure Databricks

  ,

  Unity Catalog

  , and

  Delta Lake

  .
• Strong knowledge of

  cluster configuration, compute policies, and workspace design

• Familiarity with

  Azure ecosystem

  (Azure Data Lake, Key Vault, Azure Active Directory).
• Understanding of

  data governance, security policies, and compliance standards

  (handling of

  PII/PHI

  ).
• Experience with

  CI/CD pipelines

  ,

  version control (Git)

  , and infrastructure-as-code practices is desirable.