Databricks Engineer

:

Job Title: Senior Databricks Architect / Lead Databricks Engineer

Experience:

Job Overview

Databricks Architect

Key Responsibilities

• Lead

  end-to-end Databricks architecture

  : ingestion, transformation, governance, analytics.
• Design scalable

  Lakehouse Architecture

  using Delta Tables, schema evolution, ACID transactions, time-travel, and interoperability.
• Integrate Databricks with enterprise

  ETL / orchestration tools

  .
• Build optimized pipelines with

  Spark (DataFrame API & Spark SQL)

  using Python, Scala, and SQL.
• Utilize Databricks-native features:
  

• Delta Live Tables (DLT)

• Unity Catalog

• Workflows

• Lakeflow & Lakehouse Federation

• Perform

  cluster configuration, performance tuning, cost optimization, security & monitoring

  .
• Build real-time streaming pipelines using

  Structured Streaming + Autoloader

  .
• Implement DevOps & automation:

  Terraform, GitHub Actions, Azure DevOps CI/CD

  .
• Create scalable data pipeline architectures using Databricks best practices.
  
• Strong expertise in

  data modeling

  (conceptual, logical, physical).
• Collaborate closely with CXOs, architects, and global stakeholders.
  
• Prepare solution proposals, architecture diagrams, technical documents, RFx responses, and presentations.
  
• Mentor and guide engineering teams to deliver high-quality data solutions.
  

Required Skills

• 10+ years of IT experience

  with

  4+ years hands-on in Databricks

• Deep expertise in:
  
• Lakehouse architecture & Delta ecosystem
  
• Spark DataFrame API, Spark SQL
  
• Python, Scala, SQL
  
• Unity Catalog, DLT, Workflows
  
• Structured Streaming & Autoloader
  
• CI/CD (GitHub Actions, Azure DevOps) & Terraform
  
• Strong skills in cluster performance, cost governance, data security, and monitoring
  
• Excellent communication & stakeholder management