Role & responsibilities
Architecture & Design
- Define and own the target lakehouse architecture on Azure Databricks, including medallion (bronze / silver / gold) layering, storage design on ADLS Gen2, and integration patterns with Azure Data Factory.
- Design canonical and dimensional data models for finance, pricing, and operational domains, and set modeling standards the whole team follows.
- Produce architecture decision records (ADRs) so that platform choices are documented, reviewable, and reproducible.
- Design ingestion patterns for diverse source systems including AS400 / DB2, Oracle, SQL Server, transportation management systems, and SaaS finance applications.
Data Governance with Unity Catalog
- Design and implement the Unity Catalog structure (metastores, catalogs, schemas) aligned to environments and business domains
- . Implement fine-grained access control, row- and column-level security, and data masking for sensitive data.
- Establish data ownership and stewardship, lineage, tagging, and audit practices using Unity Catalog capabilities. Forward Air | Omni Logistics
- Senior Databricks Architect Job Description
- Partner with Finance, Information Security, and business data owners to translate governance policies into enforceable platform controls.
- Define data quality rules and expectations (e.g., Delta Live Tables expectations or equivalent) and embed them into pipelines.
Hands-On Engineering
- Build and review PySpark, Spark SQL, and Delta Lake code; lead by example on code quality, testing, and documentation.
- Design orchestration using Databricks Workflows and ADF with clear dependencies, retries, and failure handling.
- Tune performance and cost: cluster policies, job compute vs. all-purpose compute, Photon, partitioning, Z-ordering / liquid clustering, and table maintenance.
- Troubleshoot complex production issues and drive root-cause fixes rather than workarounds.
Migration & Modernization
- Plan and lead the migration of workloads from legacy warehouses and semantic layers (e.g., Redshift, SSAS) onto Databricks, including validation and cutover.
- Rationalize redundant pipelines and tools into a simpler, consistent platform.
Leadership & Collaboration
- Act as the technical authority for the data platform across US and India teams; run design reviews and mentor engineers.
- Work with DevOps to ensure every pipeline is deployed through CI/CD with environment parity across Dev, QA, and Production.
- Communicate architecture choices and trade-offs clearly to technical and non-technical stakeholders.
Preferred candidate profile
- 12-15 years in data engineering and data architecture, with 4+ years of hands-on Azure Databricks in production.
- Deep, practical experience implementing Unity Catalog: catalog design, access control, lineage, and governance policies.
- Expert in PySpark, Spark SQL, and Delta Lake, including performance tuning at scale.
- Strong data modeling skills (dimensional, data vault, or canonical) for enterprise reporting and analytics.
- Experience with Azure Data Factory, ADLS Gen2, Azure Key Vault, and Entra ID integration.
- Proven record of migrating workloads from legacy data warehouses to a lakehouse.
- Working knowledge of CI/CD for data (Azure DevOps or GitHub), Git branching, and Databricks Asset Bundles.
- Excellent written and verbal communication; comfortable presenting to senior leadership.
