Azure Data Engineer

Role & responsibilities

Data Engineer

Key Responsibilities

• Design, develop, and maintain

  ETL pipelines

  using

  PySpark

  and

  Databricks

  .
• Work with large-scale datasets to ensure data quality, integrity, and availability.
• Optimize data workflows for performance and cost efficiency in cloud environments (Azure/AWS/GCP).
• Collaborate with data scientists, analysts, and business teams to deliver reliable data solutions.
• Implement best practices for

  data governance

  ,

  security

  , and

  compliance

  .
• Monitor and troubleshoot data pipelines to ensure smooth operations.
• Document technical designs, processes, and standards.

Required Skills & Qualifications

• 5+ years

  of experience in

  data engineering

  or related roles.
• Strong proficiency in

  PySpark

  and

  Databricks

  .
• Hands-on experience with

  big data technologies

  (Spark, Hadoop, Delta Lake).
• Expertise in

  SQL

  and data modeling.
• Experience with

  cloud platforms

  (Azure Data Lake, AWS S3, or GCP BigQuery).
• Knowledge of

  CI/CD pipelines

  and version control (Git).
• Familiarity with

  data warehousing

  and

  data lake architectures

  .
• Strong problem-solving and analytical skills.

Preferred Skills

• Experience with

  Airflow

  or other orchestration tools.
• Knowledge of

  streaming frameworks

  (Kafka, Spark Streaming).
• Exposure to

  machine learning pipelines

  and

  data science workflows

  .
• Understanding of

  DevOps practices

  for data engineering.