Data Engineer

Azure Data Engineer

Key Responsibilities

• Design, develop, and maintain scalable

  ETL pipelines

  for batch data ingestion and egress using Azure Data Factory, Databricks, and Spark.
• Implement data transformation workflows using

  PySpark

  and

  Scala Spark

  for high-performance data processing.
• Work extensively with

  Azure Cloud

  services including

  Azure Data Lake, Azure Data Factory, Azure Databricks, Azure SQL Database, and related services

  .
• Troubleshoot, debug, and resolve job failures by identifying root causes.
• Optimize data processing jobs for performance and cost efficiency.
• Collaborate with business and technical teams to gather requirements and translate them into effective data solutions.
• Develop reusable and efficient data engineering solutions following best practices.
• Maintain detailed documentation for solutions, processes, and troubleshooting guides.

Technical Skills Required

• Microsoft Certified: Azure Fundamentals

  (preferred)

• Microsoft Certified: Azure Data Engineer Associate

  (preferred)
• Strong programming skills in

  Python

  and

  Scala

  .
• Hands-on experience with

  Spark (PySpark & Scala Spark)

  .
• Proficiency in

  SQL

  for data querying and transformation.
• Good knowledge of

  Azure Cloud

  ,

  Azure Data Factory (ADF)

  , and

  Azure Databricks

  .
• Strong understanding of

  ETL pipeline design and implementation

  .
• Proven ability to troubleshoot jobs and analyze root causes effectively.