Data Engineer

Data Engineer

Responsibilities

• Develop fault-tolerant data pipelines running on a cluster.
• Write and optimize efficient

  SQL queries

  with

  Python

  and

  Hive

  for handling large datasets in Big-Data environments.
• Document technical design documents for given requirements or JIRA stories.
• Work closely with the overall Enterprise Data & Analytics Architect and Engineering leads to ensure adherence to best practices.
• Assure quality, security, and compliance requirements are met for the supported areas.
• Communicate results and business impacts of data initiatives to key stakeholders to collaboratively solve business problems.
• Debug, tune, and optimize

  PySpark data pipelines

  .
• Develop scalable and modular solutions.
• Coordinate with users, technical teams, and Data/Solution architects.

Required Skills & Qualifications

• 1-8 years of hands-on experience

  developing data pipelines for ingestion or transformation using

  Python (PySpark)

  /

  Spark SQL

  in

  AWS cloud

  .
• Advanced experience in writing and optimizing efficient

  SQL queries

  .
• Experience in development and processing of data at scale using technologies like

  EMR, Lambda, Glue, Athena, Redshift

  , and

  Step Functions

  .
• Experience with

  Git

  and

  CI/CD pipelines

  to deploy cloud applications.
• Strong understanding and implementation of

  PySpark data frames, joins, partitioning

  , and parallelism.
• Understanding of

  Spark UI, Event Timelines, DAG

  , and Spark config parameters for tuning pipelines.
• Experience in

  Data-Modelling

  ,

  Big data

  ,

  Hadoop

  ,

  Hive

  , and

  ETL pipelines

  .
• Familiarity with

  IaC tools

  like

  Terraform

  .
• Experience working in

  Agile implementations

  .
• Good knowledge of designing Hive tables with partitioning for performance.
• Excellent communication skills to coordinate with various stakeholders.