Databricks Expert

Job Title:

Experience:

Job Summary:

Databricks Architect

The role involves leading architecture design, ensuring performance, cost optimisation, and governance across data pipelines, and collaborating with data scientists, engineers, and business stakeholders.

Key Responsibilities:

• Architect and implement

  end-to-end data solutions

  using

  Azure Databricks / AWS Databricks / GCP Databricks

  .
• Design and develop

  data lakehouses

  , ETL/ELT pipelines, and

  data integration frameworks

  .
• Define best practices for

  data ingestion, transformation, orchestration, and governance

  .
• Lead modernisation of legacy systems into Databricks-based architectures.
  
• Optimise Databricks clusters for

  performance, scalability, and cost efficiency

  .
• Integrate Databricks with

  Azure Data Lake, Synapse, Power BI, Kafka, Snowflake, and other tools

  .
• Collaborate with cross-functional teams to define

  data modelling and data architecture standards

  .
• Implement

  CI/CD for Databricks notebooks

  using DevOps pipelines.
• Ensure

  security, compliance, and data quality

  across the data ecosystem.
• Mentor data engineers and developers on Databricks best practices.
  

Required Skills and Experience:

• 10+ years

  of overall data architecture and engineering experience.

• 7+ years

  of hands-on experience in

  Apache Spark, PySpark, and Databricks

  .
• Strong expertise in

  SQL, Python, and Delta Lake

  .
• Solid understanding of

  data lakehouse architecture and Medallion architecture

  .
• Proven experience integrating Databricks with

  Azure Data Factory, Synapse, Event Hubs, ADLS Gen2

  (or AWS/GCP equivalents).
• Hands-on experience with

  CI/CD pipelines, Git integration, and Infrastructure as Code (IaC)

  tools like Terraform or ARM.
• Experience in

  data performance tuning

  , cluster optimization, and

  monitoring

  .
• Exposure to

  MLflow, AutoML, and data science collaboration

  within Databricks is a plus.
• Excellent communication and stakeholder management skills.
  

Preferred Qualifications:

• Databricks Certified

  Data Engineer Professional

  or

  Data Architect

  .
• Experience with

  streaming data frameworks (Kafka, Spark Streaming)

  .
• Knowledge of

  Power BI / Tableau / Looker

  integration.
• Cloud certification in

  Azure, AWS, or GCP

  .

devika.raju@kumaran.com