Data Engineer (Databricks ) Senior Manager

Key Responsibilities

1. Databricks Enterprise Strategy

• Define and own the enterprise-wide Databricks platform strategy, roadmap, and adoption model in line with business goals.
• Set up a Databricks Center of Excellence (CoE), including best practices, reusable frameworks, and engineering standards.
• Assess new Databricks capabilities (Unity Catalog, Delta Live Tables, Serverless, Photon, Mosaic AI, Databricks SQL) and lead their adoption.
• Drive platform cost optimization (FinOps), workspace strategy, and capacity planning.
• Act as the primary technical liaison with Databricks and cloud vendor partners.

2. Advanced DABs (Databricks Asset Bundles) Framework

• Design and implement an enterprise-grade CI/CD framework using Databricks Asset Bundles for consistent, repeatable deployments.
• Standardize Infrastructure-as-Code and Configuration-as-Code practices across environments (Dev, QA, UAT, Prod).
• Integrate DABs with DevOps tools such as Azure DevOps, GitHub Actions, GitLab CI, Jenkins, and Terraform.
• Build reusable bundle templates, deployment patterns, and automated testing and validation pipelines.
• Ensure version control, environment promotion, and release governance across all Databricks workloads.

3. Governance Implementation

• Lead the design and rollout of data governance using Unity Catalog, covering access controls, data lineage, auditing, and data discovery.
• Implement data security, privacy, and compliance controls (e.g., GDPR, HIPAA, SOC 2, CCPA, as applicable).
• Define data quality frameworks, metadata management standards, and data ownership and stewardship models.
• Work with InfoSec, Risk, and Compliance teams to enforce policies such as row/column-level security, data masking, and fine-grained permissions.
• Set up monitoring, observability, and audit mechanisms across the data platform.

4. Data Lakehouse Architecture

• Architect and evolve the enterprise Data Lakehouse using Delta Lake and the Medallion Architecture (Bronze, Silver, Gold).
• Design scalable data models that support analytics, BI, ML/AI, and real-time use cases.
• Lead migrations from legacy data warehouses, Hadoop, and on-prem platforms to the Databricks Lakehouse.
• Ensure performance tuning, storage optimization (Z-Ordering, Liquid Clustering, partitioning), and high availability.
• Enable data sharing through Delta Sharing and support data mesh and data product approaches where relevant.

5. Cloud Architecture (AWS / Azure)

• Design secure, scalable, and resilient cloud-native data architectures on AWS and/or Azure.
• Oversee the integration of Databricks with cloud services, such as:

• Azure:

  ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Synapse, Entra ID, Purview

• AWS:

  S3, Glue, Kinesis, Lambda, IAM, Redshift, Lake Formation
• Define networking, identity, and security architecture, including VNet/VPC injection, Private Link, and SSO/SCIM.
• Drive disaster recovery, business continuity, and multi-region strategies.

6. Enterprise Data Pipeline Design

• Lead the design and delivery of robust batch and streaming pipelines using PySpark, Spark SQL, Structured Streaming, Delta Live Tables, and Auto Loader.
• Establish reusable ingestion, transformation, and orchestration frameworks (Databricks Workflows, Airflow, ADF).
• Ensure pipelines meet SLAs, scalability, reliability, and data quality requirements.
• Promote engineering excellence through code reviews, testing standards, modular design, and documentation.
• Enable real-time and near-real-time data processing for critical business use cases.

7. Portfolio Management

• Manage a portfolio of data engineering programs and projects, ensuring delivery on scope, schedule, budget, and quality.
• Prioritize initiatives based on business value, ROI, and strategic alignment.
• Own resource planning, budgeting, vendor management, and risk mitigation across the portfolio.
• Report progress, KPIs, and outcomes to senior leadership and executive stakeholders.
• Run delivery governance using Agile, SAFe, or hybrid methodologies.

People Leadership

• Lead, mentor, and grow a team of data engineers, architects, and tech leads, including onshore and offshore teams.
• Build a culture of innovation, accountability, and continuous learning.
• Drive hiring, performance management, and career development for the team.