Databricks Lead Engineer / Databricks Architects

Required Skills & Experience

• 6-9 years of experience in

  Consulting, Data, and Analytics

  , with a strong background in

  Databricks solutions

  .
• Hands-on experience in

  designing and implementing big data solutions

  , including

  data pipelines

  for large and complex datasets.
• Expertise in

  Python, PySpark

  , and orchestration frameworks such as

  Airflow, Oozie, Luigi

  , etc.
• Strong understanding of

  Databricks workspace

  and its components (workspace, compute, clusters, jobs, Unity Catalog, UC permissions).
• Ability to configure, schedule, and manage

  Databricks clusters and jobs

  , including performance optimization and monitoring.
• Experience with

  data modeling

  ,

  SQL queries, joins, stored procedures, relational schemas

  , and schema design.
• Proficiency in working with

  structured, semi-structured, and unstructured data

  (CSV, Parquet, Delta Lake, Delta Tables, Delta Live Tables).
• Strong knowledge of

  Spark architecture

  , Spark-Streaming, and analyzing jobs using

  Spark UI

  for monitoring and troubleshooting.
• Knowledge of using

  REST API endpoints

  for data consumption and integration with external sources.
• Proven experience in

  ETL / Data Warehouse transformation processes

  and building streaming/real-time data processing solutions.
• Experience in

  cloud-based data platforms

  such as

  AWS (preferred), GCP, or Azure

  , including managing data sources like AWS S3.
• Ability to design and implement

  end-to-end data ingestion pipelines

  , ensuring data quality and consistency.
• Experience with

  CI/CD pipelines configuration in Databricks

  .
• Strong logical structuring, problem-solving, verbal, written, and presentation skills.
  
• Ability to work as an

  individual contributor

  as well as lead a

  team in an Agile setup

  .
• Capable of delivering and presenting

  Proof of Concepts (POCs)

  to stakeholders.

Optional / Good to Have

• Experience in

  Databricks Administration

  (user access, Spark tuning, data frame manipulation, etc.).
• Exposure to

  real-time data movement solutions

  with security and encryption protocols.
• Experience in

  Application Development

  or

  Data Warehousing

  across enterprise technologies.

• Life Sciences industry

  domain knowledge.