Databricks Implementation Lead

• Experience needed:

  8-12 years

• Type:

  Full-Time

• Mode:

  WFO

• Shift:

  IST

• Location:

  Hyderabad, India

• Notice Period:

  Immediate to 15 days

About the Role

We are seeking an experienced

data architecture

Key Responsibilities

1. Project Leadership

• Own the complete

  Databricks implementation lifecycle

  from requirement gathering, architecture design, and environment setup to production deployment and handover.
• Act as the primary

  technical liaison

  between business stakeholders, data engineering teams, and cloud platform teams.
• Ensure alignment with project goals, timelines, and compliance standards (

  data governance, security, cost optimization

  ).

2. Architecture & Design

• Design scalable and secure

  data pipelines

  leveraging

  Databricks, Delta Lake

  , and

  cloud-native services (Azure, AWS, or GCP)

  .
• Define the

  data ingestion, transformation, and consumption architecture

  , integrating multiple data sources (APIs, databases, files, streaming).
• Establish

  CI/CD pipelines

  for Databricks notebooks and jobs, enabling automation and version control (using Git / Azure DevOps).

3. Technical Implementation

• Lead and mentor a team of

  data engineers

  to develop

  ETL/ELT pipelines

  using

  PySpark, SQL, and Databricks workflows

  .
• Configure clusters, jobs, and

  Lakehouse storage

  following best practices.
• Optimize

  Databricks compute and query performance

  for cost and efficiency.
• Implement

  data quality frameworks, monitoring dashboards, and error-handling logic

  .

4. Governance & Operations

• Set up

  access control, role-based security, and data lineage tracking

  in

  Unity Catalog

  .
• Collaborate with the

  CloudOps team

  for environment setup, monitoring, and cost governance.
• Drive

  documentation, training, and knowledge transfer

  to ensure sustainability and handover readiness.

Required Skills & Experience

• 8+ years of experience in

  data engineering or big data projects

  .
• 3+ years of

  hands-on Databricks experience

  (SQL, PySpark, Delta Lake, MLflow).
• Strong experience with

  ETL/ELT pipeline design

  ,

  data modeling

  , and

  data warehousing

  .
• Experience with

  Azure Data Factory, AWS Glue

  , or similar orchestration tools.
• Proficiency in

  Python, SQL

  , and

  Spark optimization techniques

  .
• Experience implementing

  DevOps practices (CI/CD, Git, Databricks Repos)

  .
• Familiarity with

  data governance frameworks, Unity Catalog, and RBAC

  .
• Proven track record in

  leading multi-member implementation teams

  .

Preferred Qualifications

• Databricks Certified Data Engineer Professional

  or

  Solution Architect certification

  .
• Experience with

  AI/ML workflows

  and model management using

  MLflow

  .
• Exposure to

  Power BI, Tableau, or Looker

  for downstream visualization.
• Strong communication skills and experience in

  client-facing roles

  .

Soft Skills

• Excellent

  problem-solving and analytical skills

  .
• Strong

  leadership and mentoring

  capabilities.
• Ability to work in a

  fast-paced, cross-functional environment

  .
• Clear

  documentation and presentation

  skills.