Azure Databricks

Key Responsibilities:

• Data Pipeline Development

  : Design, build, and maintain scalable

  ETL (Extract, Transform, Load)

  data pipelines using

  Azure Databricks

  ,

  Apache Spark

  , and

  Python

  .

• Spark Optimization

  : Develop and optimize

  Spark jobs

  for large-scale data processing on

  Databricks

  . Ensure that the jobs run efficiently, leveraging the capabilities of

  distributed computing

  for optimal performance.

• Data Integration

  : Integrate data from various sources, including

  structured

  and

  unstructured

  data, into the Azure cloud environment using

  Databricks

  and related tools.

• Collaboration with Data Scientists & Analysts

  : Collaborate with

  data scientists

  ,

  analysts

  , and

  business stakeholders

  to understand data requirements and deliver robust data solutions that enable advanced analytics, machine learning, and reporting.

• Azure Integration

  : Work closely with

  Azure services

  such as

  Azure Data Lake

  ,

  Azure SQL Database

  ,

  Azure Blob Storage

  ,

  Azure Synapse Analytics

  , and

  Azure Data Factory

  for comprehensive data processing solutions.

• Data Transformation

  : Use

  Spark SQL

  ,

  PySpark

  , and

  Databricks notebooks

  to perform

  data transformations

  and enable the conversion of raw data into actionable insights.

• Automation & Scheduling

  : Implement automated job scheduling and orchestration for regular data processing tasks, ensuring data is consistently processed and available for downstream consumption.

• Performance Tuning & Troubleshooting

  : Optimize the performance of data workflows and

  Spark applications

  on

  Databricks

  . Troubleshoot and resolve data-related issues and bottlenecks.

• Cloud Security

  : Ensure that

  data security

  and

  compliance standards

  are followed for cloud-based solutions, including managing data access, encryption, and auditing within the

  Azure Databricks

  environment.

• Monitoring & Logging

  : Implement logging and monitoring practices for the

  Azure Databricks

  environment to track job performance, failures, and troubleshooting efforts.

• Documentation & Best Practices

  : Maintain proper documentation for data pipelines, processes, and technical workflows. Follow best practices for coding, version control, and deployment.

• Stay Updated with Technology Trends

  : Keep up to date with the latest developments in

  Azure Databricks

  ,

  Apache Spark

  , and related technologies. Apply new techniques to improve performance and scalability.

Required Qualifications & Skills:

• 3-5 years

  of hands-on experience in

  data engineering

  and working with

  Azure Databricks

  .
• Strong proficiency in

  Apache Spark

  , particularly in

  Databricks

  for building large-scale data pipelines and distributed data processing applications.
• Solid experience with

  Azure cloud services

  , including

  Azure Data Lake

  ,

  Azure SQL Database

  ,

  Azure Blob Storage

  ,

  Azure Synapse

  , and

  Azure Data Factory

  .
• Proficiency in

  Python

  ,

  Scala

  , or

  SQL

  for data engineering tasks, with a focus on

  PySpark

  for data processing.
• Experience working with

  structured

  and

  unstructured data

  from a variety of sources, including

  relational databases

  ,

  APIs

  , and

  flat files

  .
• Familiarity with

  Databricks notebooks

  for developing, testing, and sharing data workflows, and using them for collaboration.
• In-depth understanding of

  ETL processes

  ,

  data pipelines

  , and

  data transformation

  techniques.
• Hands-on experience with

  cloud-based data storage

  solutions (e.g.,

  Azure Data Lake

  ,

  Blob Storage

  ) and data warehousing concepts.
• Knowledge of

  data security

  best practices in a cloud environment (e.g.,

  data encryption

  ,

  access controls

  ,

  Azure Active Directory

  ).
• Experience with

  CI/CD

  pipelines and version control systems like

  Git

  .
• Familiarity with

  containerization

  and deployment practices using

  Docker

  and

  Kubernetes

  is a plus.
• Strong debugging, performance tuning, and problem-solving skills.
• Excellent written and verbal communication skills, with the ability to collaborate effectively across teams.

• Bachelor's degree

  in

  Computer Science

  ,

  Information Technology

  , or a related field.