Senior Databricks Engineer

Senior Databricks Engineer

Exp-9-15 years

Work location- Bangalore

We are looking for a Senior Databricks Engineer to become a cornerstone of our Data & Analytics team. In this role, you will own the design and delivery of our Databricks-first data platform - building high-performance data pipelines, architecting scalable Lakehouse solutions, and championing engineering best practices across the team. You will work closely with business analysts, a data modeler and business stakeholders to turn raw data into reliable, production-grade analytics products.

Responsibilities

• Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
• Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
• Unity Catalog: Govern data assets - tables, volumes, and models - through Unity Catalog, enforcing fine-grained access control and lineage tracking.
• Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
• Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
• ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
• Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
• Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
• Code
• Write clean,
• and
• coding standards.
• CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
• Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
• Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
• Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
• Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
• Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
• Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
• Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
• Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.

Requirements

• 10+
• Azure + Databricks Certified (Mandatory)
• Databricks Certified Data Engineer Professional or Associate (mandatory).
• 10+ years of hands-on experience in Data Engineering roles.
• Deep expertise in Databricks - Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
• Advanced PySpark and Python skills for large-scale distributed data processing.
• Strong SQL skills: query authoring, performance tuning, and data modeling.
• Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
• Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
• Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
• Experience with Structured Streaming or other real-time data processing frameworks.
• Experience with Microsoft Purview for data cataloguing and governance.
• Familiarity with dbt (data build tool) for SQL-based transformation workflows.
• Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
• Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
• Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
• Background in data mesh or data product thinking.
• Databricks,
• PySpark, Python, SQL
• Azure
• DevOps
• Azure
• Terraform, Bicep, DABs

Skills: Databricks , PySpark , Python