Principal Engineer - Databricks

Flexera Software

12 - 17 years

50 - 55 Lacs

bengaluru

Posted:1 day ago| Platform:

Apply

Skills Required

fault tolerance azure data lake scala production microsoft azure distribution system pyspark networking performance engineering engineering data engineering sql data bricks iam grafana kafka integration software engineering aws communication skills

Work Mode

Work from Office

Job Type

Full Time

Job Description

Architect, design, and implement scalable and performant Databricks-based data pipelines, including batch and streaming workloads.

Optimize Databricks Lakehouse architecture including Delta Lake, Unity Catalog, and Photon for cost efficiency, data reliability, and query performance at scale.
Define and implement advanced data models and ontologies to support complex analytics and AI/ML workflows.
Design and tune complex Spark applications (PySpark and/or Scala) for high-volume data processing, including structured streaming and incremental processing.
Implement orchestration workflows (e.g., Databricks Workflows, Azure Data Factory, or similar) to automate data pipelines with robust monitoring and alerting.
Drive best practices for data governance, security, and lineage, leveraging Unity Catalog, table ACLs, and audit capabilities.
Optimize cluster configurations, autoscaling strategies, and cost management within Databricks environments.
Collaborate with data scientists and analytics engineers to enable self-service access to high-quality, well-documented datasets and features.
Lead complex troubleshooting efforts, deep performance tuning, and root-cause analysis across distributed systems.
Author and maintain technical artifacts, including architecture diagrams, design docs, reference implementations, and best practice guides.
Mentor and coach engineers across teams on Databricks internals, Spark performance tuning, and modern data engineering architectures.
Stay on top of emerging Databricks features (e.g., Mosaic AI, serverless compute, Delta Live Tables), advocating for their adoption when they bring tangible value.

Youll be expected to have:

Bachelor s or higher degree in Computer Science, Software Engineering, or a related field.
12+ years of software engineering experience, with 6+ years specifically in building large-scale data platforms.

Expert-level proficiency in Databricks, including:

Delta Lake internals (ACID transactions, time travel, Z-ordering, data compaction strategies)
Unity Catalog for data governance, security, and lineage
Advanced Spark tuning (shuffle management, partitioning strategies, caching, cluster configuration)
Databricks SQL Warehousing, Workflows, and MLflow integration
Strong expertise with cloud-native data engineering in Azure or AWS, including storage (ADLS/S3), networking, and IAM integration.
Knowledge of streaming architectures (Apache Kafka, structured streaming), schema evolution, and event-driven data modeling.
Hands-on experience with metadata catalogs, orchestration frameworks, and observability tooling (e.g., Datadog, Grafana).
Strong understanding of distributed systems, fault tolerance, and large-scale performance engineering.
Proven ability to solve complex, ambiguous problems and make architectural trade-offs in real-world production systems.
Excellent technical communication skills ability to document, present, and explain technical concepts clearly to varied audiences.
Prior experience mentoring engineers, conducting design reviews, and setting technical direction across teams.
A continuous learner mindset actively keeping pace with evolving Databricks platform capabilities and broader data engineering trends.

More Jobs at Flexera Software

Analyst - Renewals

Bengaluru

2 - 7 yrs

INR 5 - 10 Lacs

Senior Software Licensing Consultant

Bengaluru

4.0 - 8.0 yrs

INR 12 - 17 Lacs

Senior Content Engineer

Bengaluru

4.0 - 9.0 yrs

INR 10 - 11 Lacs

Senior Director, Cloud Engineering

Bengaluru

12.0 - 18.0 yrs

INR 35 - 40 Lacs

Application Security Engineer

Bengaluru

3.0 - 7.0 yrs

INR 8 - 13 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.