Databricks Engineer

Job Descritpion of Databricks Engineer

6 to 8 Years Relevant Experience

We are looking for an experienced Databricks Engineer to join our data engineering team and contribute to designing and implementing scalable data solutions on the Azure platform. This role involves working closely with cross-functional teams to build high-performance data pipelines and maintain a modern Lakehouse architecture.

Key Responsibilities:

• Design and develop scalable data pipelines using Spark-SQL and PySpark in Azure Databricks.
• Build and maintain Lakehouse architecture using Azure Data Lake Storage (ADLS) and Databricks.
• Perform comprehensive data preparation tasks, including:
  • Data cleaning and normalization
  • Deduplication
  • Type conversions
• Collaborate with the DevOps team to deploy and manage solutions in production environments.
• Partner with Data Science and Business Intelligence teams to share insights, align on best practices, and drive innovation.
• Support change management through training, communication, and documentation during upgrades, data migrations, and system changes.

Required Qualifications:

• 5+ years of IT experience with strong exposure to cloud technologies, particularly in Microsoft Azure.
• Hands-on experience with:
  • Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS)
  • Programming with PySpark, Python, and SQL
• Solid understanding of data engineering concepts, data modeling, and data processing frameworks.
• Ability to work effectively in distributed, remote teams.
• Excellent communication skills in English (both written and verbal).

Preferred Skills:

• Strong working knowledge of distributed computing frameworks, especially Apache Spark and Databricks.
• Experience with Delta Lake and Lakehouse architecture principles.
• Familiarity with data tools and libraries such as Pandas, Spark-SQL, and PySpark.
• Exposure to on-premise databases such as SQL Server, Oracle, etc.
• Experience with version control tools (e.g., Git) and DevOps practices including CI/CD pipelines.

Required Skills for Databricks Engineer Job

Spark-SQL and PySpark in Azure Databricks

Python

SQL