Lead Consultant-Databricks developer

Lead Consultant-

In this role, the Databricks Developer is responsible for solving the real world cutting edge problem to meet both functional and non-functional requirements.

Responsibilities

• Design and develop scalable data pipelines using Databricks (PySpark/SQL/Delta Live Tables).

• Implement ETL/ELT frameworks leveraging the Lakehouse (Bronze, Silver, Gold) architecture.

• Implement and manage data governance using Unity Catalog to ensure secure access, compliance, and centralized management of data, users, and permissions across the Databricks Lakehouse.

• Optimize data models, queries, and workflows for scalability, cost, and performance.

• Act as Databricks SME and provide guidance on best practices, governance, and security.

• Mentor junior developers, review code, and enforce coding standards.

• Integrate Databricks with cloud storage, APIs, warehouses, and BI tools.

• Implement orchestration using ADF, Airflow, Step Functions, or Databricks Workflows.

• Build reusable accelerators and frameworks for ingestion, transformation, and monitoring.

• Enable data quality, validation, and reconciliation (using tools like Great Expectations or custom).

• Set up monitoring, logging, and alerting dashboards for pipeline health.

• Collaborate with business stakeholders, architects, and analysts to deliver solutions.

• Support migration of legacy pipelines into Databricks Lakehouse.

• Contribute to architectural decisions, POCs, and innovation initiatives.

Qualifications we seek in you!

Minimum qualifications

• Bachelor's degree (CS, CE, CIS, IS, MIS, Engineering) or equivalent work experience.

• Experience in data engineering with at least Databricks experience.

• End-to-end implementation of at least 2 Databricks projects(migration/integration).

• Strong background in batch and streaming data pipelines.

• Proficiency in Python (preferred) or Scala for Spark-based development.

• Expertise in SQL & Spark-SQL, data structures, and algorithms.

• Deep knowledge of Databricks components: Delta Lake, DLT, dbConnect, REST API 2.0, Workflows orchestration.

• Strong in performance optimization for pipelines (efficiency, scalability, cost reduction).

• Hands-on experience with Apache Spark, Hive, and Lakehouse architecture.

• Cloud expertise (Azure/AWS) includes storage (ADLS/S3), messaging (ASB/SQS), compute (ADF/Lambda), and databases (CosmosDB/DynamoDB/Cloud SQL).

• Experience writing unit tests and integration tests for data pipelines.

• Ability to work with architects and lead engineers to design solutions meeting functional & non-functional requirements.

• Team player with experience in teams of 5+ engineers.

• Strong communication and client-facing skills.

• Keeps updated with emerging technologies and industry trends.

• Strong analytical and problem-solving abilities.

• Positive attitude towards continuous learning and upskilling

• Good to have Databricks SQL Endpoint understanding.

• Good to have understanding on LakeflowConnect, Lakeflow Declarative Pipelines

• Good To have CI/CD experience to build the pipeline for Databricks jobs.

• Good to have if worked on migration project to build Unified data platform.

• Good to have knowledge of DBT.

• Good to have knowledge of docker and Kubernetes.

• Certification on Databricks Associate level.

• Any one Cloud Certification (AWS/Azure) Practitioner or Associate Level