Senior Data Engineer

Job Description

Senior Data Engineer

Responsibilities:

• Design, develop, and optimize scalable data pipelines using Databricks in

  AWS

  such as Glue, S3, Lambda, EMR, Databricks notebooks, workflows and jobs.
• Building data lake in WS Databricks.
• Build and maintain robust

  ETL/ELT workflows

  using

  Python

  and

  SQL

  to handle structured and semi-structured data.
• Develop distributed data processing solutions using

  Apache Spark

  or

  PySpark

  .
• Partner with data scientists and analysts to provide high-quality, accessible, and well-structured data.
• Ensure data quality, governance, security, and compliance across pipelines and data stores.
• Monitor, troubleshoot, and improve the performance of data systems and pipelines.
• Participate in code reviews and help establish engineering best practices.
• Mentor junior data engineers and support their technical development.

Requirements

• Bachelor's or master's degree in computer science, Engineering, or a related field.
• 5+ years of hands-on experience in

  data engineering

  , with at least 2 years working with

  AWS

  Databricks

  .
• Strong programming skills in

  Python

  for data processing and automation.
• Advanced proficiency in

  SQL

  for querying and transforming large datasets.
• Deep experience with

  Apache Spark/PySpark

  in a distributed computing environment.
• Solid understanding of data modelling, warehousing, and performance optimization techniques.
• Proficiency with AWS services such as

  Glue

  ,

  S3

  ,

  Lambda

  and

  EMR

  .
• Experience with version control Git or Code commit
• Experience in any workflow orchestration like Airflow, AWS Step funtions is a plus