Databricks Lead

Name of Position:

Role Overview:

Key Responsibilities

• Lead end-to-end

  data engineering and modernization projects

  using Azure Databricks and related technologies.
• Design, develop, and maintain scalable

  data pipelines and workflows

  using

  Azure Data Factory

  and

  PySpark

  .
• Work closely with cross-functional teams to define and implement

  data ingestion, transformation, and integration

  processes.
• Drive

  data migration and cloud modernization

  strategies, ensuring scalability, performance, and security.
• Build and maintain

  CI/CD pipelines

  for data solutions using

  Azure DevOps

  .
• Manage project deliverables, ensuring adherence to

  Agile methodologies

  and best practices.
• Conduct

  technical workshops

  with business and technical stakeholders to define requirements and implementation strategies.
• Ensure compliance with

  data security, governance, and quality standards

  .
• Mentor team members and foster a collaborative technical environment.
  

Required Skills & Experience

• Education:

  Bachelors degree in Computer Science, Information Systems, Data Engineering, or related field.

• Experience:

  8 - 13 years total experience in data engineering or cloud data solutions.

• Technical Expertise:

• 8+ years working with

  Azure Databricks, Azure Data Factory, and PySpark

  .
• 5+ years of hands-on experience in

  Microsoft Azure

  .
• 3+ years of experience using

  Python

  for data extraction from unstructured sources (XML, PDF, HTML).
• Strong understanding of

  Spark architecture

  and performance optimization.
• Experience with

  Azure DevOps

  for CI/CD, deployment, and builds.
• Proficiency in

  data integration, ELT, data wrangling, and messaging patterns

  .
• Working knowledge of

  data security, EAI, and cloud computing

  concepts.

• Certifications:

  Databricks certification preferred.

• Methodologies:

  Experience in

  Agile software development

  environments.

Preferred Candidate Profile

• Strong leadership and project management skills.
  
• Excellent communication, organization, and analytical thinking abilities.
  
• Ability to drive end-to-end cloud data projects and lead technical discussions.
  
• Passionate about modern data engineering practices and emerging cloud technologies.
  

Perks and Benefits

• Competitive compensation package
  
• Health insurance coverage
  
• Continuous learning & certification support
  
• Dynamic and collaborative work environment
  
• Performance-based incentives
  

Employment Details

• Industry Type:

  IT Services & Consulting

• Department:

  Data Engineering / Cloud Solutions

• Employment Type:

  Full Time, Permanent

• Role Category:

  Data & Cloud Engineering