Senior Principal Consultant- Databricks Platform Architect!

Senior Principal Consultant-

We are seeking a highly skilled and experienced Databricks Platform Architect to lead the design, implementation, and optimization of our Databricks platform. The ideal candidate will have a strong background in cloud-based data processing systems, data warehousing, and big data technologies. They will work closely with our data engineering team to ensure that our Databricks platform is optimized for performance, scalability, and reliability.

Responsibilities

• Architect and design end-to-end Databricks-based solutions covering ingestion, transformation, governance, and consumption.

• Lead platform architecture initiatives including Unity Catalog, Delta Lake, and advanced Databricks capabilities.

• Define and enforce architecture standards, security, and governance frameworks.

• Provide technical direction, oversight, and best practices for Databricks platform implementation.

• Optimize performance, scalability, and cost-efficiency of Databricks environments.

• Guide engineering teams on Databricks, Spark, and cloud-native services.

• Work closely with data engineering, platform, and business teams to ensure seamless integration with enterprise systems.

• Provide technical governance, risk management, and quality reviews during solution delivery.

• Act as a trusted advisor to clients, driving business value from data engineering initiatives.

• Stay ahead of Databricks product evolution and industry trends to recommend relevant use cases.

• Contribute to accelerators, frameworks, reusable assets, and best practices for Databricks delivery.

• Publish whitepapers, blogs, and thought leadership content on cloud data engineering.

• Support pre-sales, proposals, and client engagements, including proof of concepts and demos.

• Build strategic executive relationships and help expand data platform adoption within accounts.

• Mentor and develop talent, enabling certification, training, and upskilling in Databricks and cloud data engineering.

Qualifications we seek in you!

Minimum qualifications

• Bachelor's Degree or equivalency (CS, CE, CIS, IS, MIS, or engineering discipline) or equivalent work experience.

• data engineering with at least hands-on Databricks experience.

• Track record of delivering 2-3 end-to-end Databricks implementations at enterprise scale.

• Proven experience as a Data/Platform Architect with hands-on Databricks expertise.

• Strong background in cloud data engineering (Azure, AWS, or GCP) with hands-on design/implementation.

• Expertise in Apache Spark, Delta Lake, SQL Warehouse, and Databricks Unity Catalog.

• Strong knowledge of Python, SQL, and data engineering pipelines experience with DBT is a plus.

• Experience in data governance, security (RBAC, SSO, encryption, compliance), and monitoring frameworks.

• Design, develop, and optimize large-scale batch pipelines for ingestion, transformation, and analytics using Databricks.

• Build and manage low-latency streaming pipelines using Spark Structured Streaming, Delta Live Tables, or other Databricks-native frameworks to enable real-time insights and decision-making.

• Solid experience in designing scalable, secure, and cost-optimized cloud-native platforms.

• Experience implementing CI/CD pipelines, DevOps practices, and Infrastructure-as-Code (Terraform, etc.).

• Excellent communication, leadership, and stakeholder management skills.

• Knowledge of Databricks workspace management, UCX migration tools, and composable architecture.

• Strong understanding of observability, monitoring, and optimization in Databricks.

• Exposure to multi-cloud architectures and integration with enterprise applications.

• Familiarity with Docker, Kubernetes, and API integrations.

• Experience in data modernization and migration projects to unified data platforms.

• Contribution to industry thought leadership (whitepapers, blogs, conferences, etc.).

• Certification on Databricks Associate or Professional level.

• Any one Cloud Certification (AWS/Azure) Associate/Professional level -Data Engineer or Architect level