Senior Data Engineer

Role Overview

Databricks

Key Responsibilities

For Databricks Engineer (6-9 Years)

• Design, develop, and maintain

  ETL/ELT pipelines

  using Databricks (Python/Scala/Spark SQL).
• Work extensively on

  Delta Lake

  , including ACID transactions, schema evolution, time-travel, and performance optimizations.
• Build and optimize workflows using

  Databricks Workflows

  ,

  DLT

  , and

  Unity Catalog

  .
• Configure and optimize

  Databricks clusters

  and Spark jobs for performance and cost efficiency.
• Work with cloud storage systems (S3/ADLS/GCS), IAM/RBAC, and networking/security components.
• Implement Medallion Architecture (Bronze Silver Gold).

For Associate Manager - Data Engineering (8-10 Years)

plus

• Lead small teams in delivering Databricks-based data solutions.
• Own end-to-end design of data pipelines and distributed systems.
• Implement DevOps/CI-CD practices using Git, Azure DevOps, Terraform, Databricks CLI.
• Ensure data governance compliance, security controls, and best engineering practices.

For Manager - Data Engineering (10-12 Years)

plus

• Drive architectural decision-making for large-scale data platforms on Databricks.
• Partner with business, product, analytics, and data science teams to align solutions with business goals.
• Lead multiple project teams, mentor engineers, and ensure delivery excellence.
• Define long-term data engineering roadmap, standards, and best practices.

Core Technical Skills (All Roles)

• Databricks Platform Expertise

  : Workspace, notebooks, DLT, Workflows, Unity Catalog.

• Apache Spark

  : PySpark, Spark SQL, Scala/Java for performance tuning.

• Delta Lake

  : ACID transactions, schema enforcement, Z-ordering, optimization.

• Programming

  : Python and/or Scala; SQL for analytics and data validation.

• Cloud Platforms

  : AWS / Azure / GCP -- storage, IAM, networking, security.

• ETL/Data Architecture

  : Batch & streaming pipelines, Medallion Architecture.

• Performance Optimization

  : Debugging data skew, memory issues, cluster tuning.

• DevOps & CI/CD

  : Azure DevOps, Git, Terraform, Databricks CLI.

• Security & Governance

  : Row-level/column-level security, encryption, access controls.

Soft Skills & Leadership Competencies

• Strong analytical and problem-solving ability.
• Effective communication with stakeholders, analysts, and cross-functional teams.
• Ability to mentor juniors and enable team growth (Associate Manager/Manager).
• Strategic thinking with the ability to influence design and technology decisions.
• Ownership mindset with strong execution focus.

Preferred Certifications

• Databricks Data Engineer Associate / Professional
• Databricks Lakehouse Fundamentals
• Cloud certifications: AWS/Azure/GCP (Associate or above)

Core Technical Skills (All Roles)

• Databricks Platform Expertise

  : Workspace, notebooks, DLT, Workflows, Unity Catalog.

• Apache Spark

  : PySpark, Spark SQL, Scala/Java for performance tuning.

• Delta Lake

  : ACID transactions, schema enforcement, Z-ordering, optimization.

• Programming

  : Python and/or Scala; SQL for analytics and data validation.

• Cloud Platforms

  : AWS / Azure / GCP -- storage, IAM, networking, security.

• ETL/Data Architecture

  : Batch & streaming pipelines, Medallion Architecture.

• Performance Optimization

  : Debugging data skew, memory issues, cluster tuning.

• DevOps & CI/CD

  : Azure DevOps, Git, Terraform, Databricks CLI.

• Security & Governance

  : Row-level/column-level security, encryption, access controls.