Data Engineer

Role Overview

Data Engineer

Key Responsibilities

• Design and implement

  real-time (Kafka, Spark/Flink)

  and

  batch (Airflow, Spark)

  pipelines for high-throughput data ingestion, processing, and transformation.


• Develop data models and manage

  data lakes and warehouses

  (Delta Lake, Iceberg, etc) to support both analytical and ML workloads.


• Integrate data from diverse sources: IoT sensors, databases (SQL/NoSQL), REST APIs, and flat files.


• Ensure pipeline

  scalability, observability

  , and

  data quality

  through monitoring, alerting, validation, and lineage tracking.


• Collaborate with AI/ML teams to

  provision clean and ML-ready datasets

  for training and inference.


• Deploy, optimize, and manage pipelines and data infrastructure across

  on-premise and hybrid environments

  .


• Participate in architectural decisions to ensure

  resilient, cost-effective, and secure data flows

  .


• Contribute to infrastructure-as-code and automation for data deployment using Terraform, Ansible, or similar tools.

Qualifications & Required Skills

• Bachelor s or Master s in Computer Science, Engineering, or related field.


• 6+ years in data engineering roles, with at least 2 years handling real-time or streaming pipelines.


• Strong programming skills in

  Python/Java

  and

  SQL

  .


• Experience with

  Apache Kafka

  ,

  Apache Spark

  , or

  Apache Flink

  for real-time and batch processing.


• Hands-on with

  Airflow

  ,

  dbt

  , or other orchestration tools.


• Familiarity with

  data modeling

  (OLAP/OLTP), schema evolution, and format handling (Parquet, Avro, ORC).


• Experience with hybrid/on-prem and cloud platforms (

  AWS/GCP/Azure

  ) deployments.


• Proficient in working with

  data lakes/warehouses

  like Snowflake, BigQuery, Redshift, or Delta Lake.


• Knowledge of

  DevOps practices

  , Docker/Kubernetes, Terraform or Ansible.


• Exposure to

  data observability, data cataloging, and quality tools

  (e.g., Great Expectations, OpenMetadata).

Good-to-Have

• Experience with

  time-series databases

  (e.g., InfluxDB, TimescaleDB) and sensor data.


• Prior experience in domains such as aviation, manufacturing, or logistics is a plus.