Data Engineer, AWS, Hadoop

Job Title: Data Engineer AWS + Hadoop

Location:

About the Role

Data Engineer

Key Responsibilities

• Design and implement

  scalable ETL/ELT pipelines

  for batch and streaming workloads.
• Build data ingestion frameworks using

  Kafka/Kinesis

  , and process data with

  Spark

  (PySpark/Scala).
• Develop and optimize

  data lakes

  and

  data warehouses

  on

  AWS

  (S3, Glue, EMR, Athena, Redshift).
• Manage and tune

  Hadoop ecosystem

  components (HDFS, Hive, Spark, Oozie/Airflow, Sqoop).
• Model data (star/snowflake), manage

  schemas, partitioning, and metadata

  ; ensure data quality (DQ checks).
• Implement

  data governance, security

  , and

  access controls

  (IAM, Lake Formation, encryption, key management).
• Set up

  orchestrations

  and

  CI/CD

  for data jobs (Airflow/AWS Step Functions, Jenkins/GitHub Actions).
• Monitor pipelines and optimize

  cost, performance, and reliability

  (CloudWatch, logs, metrics).
• Collaborate with Analytics/ML/BI teams; provide high-quality curated datasets and

  APIs/Views

  .
• Document solutions, conduct

  code reviews

  , and enforce engineering best practices.

Required Skills & Qualifications

• 7+ years

  in Data Engineering with large-scale

  distributed data

  systems.
• Strong experience with

  AWS data stack

  :

  S3, Glue, EMR, Athena, Lambda, Redshift, IAM, CloudWatch

  .
• Hands-on with

  Hadoop ecosystem

  :

  HDFS, Hive, Spark (PySpark/Scala), Kafka

  , Oozie/Airflow.
• Expertise in

  SQL

  (complex queries, performance tuning) and

  data modeling

  .
• Practical knowledge of

  streaming

  (Kafka/Kinesis, Spark Streaming/Structured Streaming).
• Experience with

  Python

  or

  Scala

  for data pipelines;

  Shell scripting

  .
• Familiarity with

  Orchestration

  (Airflow/AWS Step Functions) and

  CI/CD

  for data jobs.
• Strong understanding of

  security & governance

  (encryption, PII handling, RBAC, Lake Formation).
• Proficient with

  version control

  (Git) and

  containers

  (Docker) for reproducible jobs.
• Excellent problem-solving, communication, and collaboration skills.