Sr. Data Engineer (Databricks)

Databricks Capability in India

Role Overview:

Senior Databricks Data Engineer

Responsibilities:

• Solution Ownership:

  Take end-to-end responsibility for project delivery, including planning, prioritization, and ensuring data solutions meet quality and performance expectations.

• Stakeholder Collaboration:

  Engage with business and product teams to understand data needs, translate requirements, and align engineering solutions with strategic goals.

• Cross-Functional Communication

  : Present complex data concepts in a clear, actionable manner to non-technical stakeholders and leadership.

• Continuous Improvement:

  Identify process inefficiencies and lead initiatives to improve data workflows, scalability, and team productivity.
• Design and build scalable, reliable data pipelines using Databricks Auto Loader, Delta Live Tables (DLT), and Workflows.
• Develop and maintain curated Medallion architecture layers (Bronze, Silver, Gold) using Delta Lake and Databricks SQL.
• Orchestrate and deploy pipelines using Databricks Asset Bundles (DABs) and integrate with CI/CD tools (GitHub Actions, Azure DevOps).
• Implement data ingestion using Lakeflow Connect and integrate with external sources such as Kafka, ADLS, and REST APIs.
• Develop real-time and batch processing jobs with PySpark, SQL, and Delta Lake.
• Ensure data quality, lineage, and governance using Unity Catalog and DLT Expectations.
• Collaborate with data scientists and ML engineers to prepare feature-ready datasets for MLflow and AI workloads.
• Optimise Databricks clusters and jobs for cost, performance, and reliability.
• Participate in client workshops, provide architectural input, and communicate technical outcomes effectively to business stakeholders.
• Take ownership of projects ensuring exceptional customer outcomes
• Contribute to a data-driven culture that fosters innovation and agility.

Required Skills & Competencies

3 - 5 years of hands-on experience with Databricks

• Demonstrated experience in

  client-facing delivery

  , including conducting online workshops, eliciting data requirements, and presenting technical findings to business and technical stakeholders.
• Databricks Certification (Data Engineer Associate or Professional) required.
• PySpark, SQL, and data modeling (Dimensional, ELT).
• Medallion architecture and Delta Lake optimization (Z-Ordering, Optimize, Compaction).
• Unity Catalog, Workflows, and Feature Store.
• Azure Data Services (ADLS, ADF, Synapse) or equivalent on AWS/GCP.

• Auto Loader

  for incremental ingestion

• Delta Live Tables (DLT)

  for declarative ETL pipelines

• Lakeflow Connect

  for source integration

• Databricks SQL

  for analytics and transformation
• Ability to thrive in a fast-paced, entrepreneurial, team environment

Preferred (Nice to have)

• Prior experience in

  Retail, CPG, or Transportation & Logistics

  domains.

• CI/CD skills for Databricks

  using GitHub Actions or Azure DevOps and Databricks Asset Bundles (DABs) for deployment automation
• Experience implementing

  data quality monitoring

  and

  governance frameworks

  .

What we offer

youthful, forward-looking Data & AI brand

technical depth, consulting exposure, and personal growth

• Flexible Work Options

  – Work

  100% remotely

  or in a

  hybrid mode

  from our Chennai base — we trust our people to deliver outcomes, not fill timesheets.

• Learning & Mentorship Culture

  – Work directly with senior data and AI leaders who invest in your

  career growth, leadership development, and technical mastery

  .

• High-Calibre Clients & Projects

  – Collaborate with

  mature global clients

  pursuing ambitious

  Data and AI transformation goals

  — exposure to real enterprise-grade challenges.