Databricks Engineer

• Develop and maintain scalable data pipelines using

  PySpark

  on

  Databricks

  .
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Optimize and troubleshoot existing PySpark scripts and workflows for performance and reliability.
• Implement data quality checks, validations, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Work with cloud platforms (preferably

  Azure

  ) to manage data storage and compute resources.
• Ensure adherence to data governance and security policies.
• Document technical solutions, processes, and best practices.

Required Skills:

• 3-5 years of experience in

  data engineering

  roles.
• Strong hands-on experience with

  PySpark

  and

  Databricks

  .
• Proficiency in

  SQL

  and working with large-scale datasets.
• Experience with

  Delta Lake

  ,

  Apache Spark

  , and

  data lake architectures

  .
• Familiarity with

  CI/CD pipelines

  , version control (Git), and DevOps practices.
• Experience with

  cloud platforms

  (Azure/AWS/GCP), preferably Azure.
• Strong problem-solving and communication skills.

Preferred Qualifications:

• Experience with

  ETL tools

  and

  workflow orchestration

  (e.g., Airflow, Azure Data Factory).
• Knowledge of

  data modeling

  and

  data warehousing

  concepts.
• Exposure to

  ML/AI workflows

  and integration with data pipelines.
• Databricks certification is a plus.