Data Engineer

Role & responsibilities

Data Pipeline Development

• Design, build, and maintain scalable

  ETL/ELT pipelines

  for batch and streaming data.
• Develop real-time data ingestion frameworks using

  Apache Kafka

  (topics, partitions, consumer groups).
• Build large-scale distributed processing jobs using

  Spark / PySpark

  .

Big Data Engineering

• Work with

  Hadoop, HDFS, Hive

  for storing and processing big datasets.
• Create and optimize Hive tables (managed/external, ORC/Parquet).
  
• Implement performance tuning in Hive (partitioning, bucketing).
  

Data Transformation & Processing

• Develop complex data transformation logics using

  PySpark & Python

  .
• Optimize Spark jobs (shuffle, caching, broadcast joins, partitioning).
  

ETL Development

• Work on end-to-end ETL workflow creation, scheduling, and monitoring.
  
• Integrate data from various sources (databases, APIs, cloud storage).
  

Streaming & Real-Time Use Cases

• Build low-latency pipelines using

  Kafka Streams / Spark Structured Streaming

  .
• Manage Kafka clusters, schemas, message retention, and throughput tuning.
  

Data Quality & Reliability

• Implement automated data validation and quality checks.
  
• Develop logging, monitoring, and alerting for pipelines.
  

Collaboration & Documentation

• Work closely with Data Science, Analytics, and Business teams.
  
• Document data flows, pipeline logic, architecture, and best practices.
  

Required Skills

• Strong hands-on experience in

  Apache Spark & PySpark

• Proficiency in

  Python

  for data engineering
• Experience with

  Apache Kafka

  (producers/consumers/streams)
• Knowledge of

  Hadoop, Hive, and HDFS

• Strong skills in

  ETL development

• Advanced

  SQL

  skills
• Experience with Git, CI/CD, and workflow schedulers (Airflow/Oozie) is a plus
  
• Cloud experience (AWS/Azure/GCP) preferred
  

Good to Have

• Knowledge of Delta Lake / Iceberg / Hudi
  
• Experience with NoSQL databases (HBase / Cassandra)
  
• Understanding of data warehousing concepts