Senior Data Engineer

Role

Experience

Location

About 1DigitalStack.ai

1DigitalStack.ai combines AI and deep eCommerce data to help global brands grow faster on online marketplaces. Our platforms deliver advanced analytics, actionable intelligence, and media

automation — enabling brands to optimize visibility, efficiency, and sales performance at scale.

We partner with India’s top consumer companies — Unilever, Marico, Coca-Cola, Tata Consumer, Dabur, and Unicharm — across 125+ marketplaces globally.

Backed by leading venture investors and powered by a 220+ member team, we’re in our $5–10M

growth journey, scaling rapidly across categories and geographies to redefine how brands win on

digital shelves.

About Role

This is a high-impact, hands-on engineering role owning the core data systems that power our

analytics, AI, and automation stack.

You’ll work closely with the CTO and Engineering Leads and independently manage large,

high-throughput data pipelines that process millions of events.

Responsibilities :

● Build and maintain high-throughput, real-time data pipelines using Kafka/Pulsar with Spark, Flink, and distributed compute engines.

● Design fault-tolerant systems with zero-data-loss principles — checkpointing, replay logic, DLQs, deduplication, and back-pressure handling.

● Implement data observability — quality checks, SLA alerts, anomaly detection, lineage, and metadata insights.

● Design and manage Iceberg-based lakehouse tables (Polaris/Gravitino catalogs, schema evolution, compaction).

● Build fast OLAP layers using ClickHouse / StarRocks.

● Model data across bronze → silver → gold layers for downstream teams.

● Migrate and modernize legacy pipelines into scalable, distributed workflows.

● Orchestrate ETL workloads using Airflow, DBT, Dagster, SQLMesh.

● Optimize SQL transformations and distributed execution across Trino/Spark.

● Ensure strict security and governance across all data layers — access control, encryption,

auditability.

● Collaborate with backend, analytics, and platform teams for seamless data delivery.

Requirements

Core Technical Skills

● Extremely strong SQL — window functions, query planning, optimization.

● High comfort working with distributed & parallel workloads.

● Hands-on experience with some-many of these technologies : Apache Spark, Apache Flink, Trino, Apache Kafka, Apache Pulsar, Apache Beam

● Advanced experience in Python (preferred) or Java (strong fundamentals).

● Strong understanding of Parquet, Apache Iceberg, and Iceberg REST catalogs (Polaris / Gravitino).

● Experience with OLAP databases — ClickHouse / StarRocks.

● Experience with semantic layers — Cube.js or similar.

● Strong experience building pipelines with Airflow, DBT, Dagster, SQLMesh.

Foundational Strengths

● Solid understanding of data structures & algorithms — sorting, searching, memory models.

● Strong grasp of OLTP vs OLAP, indexing, query execution, and storage formats.

● Ability to debug distributed systems end-to-end (compute, storage, network, orchestration).

● Familiarity with cloud environments, containerization (Docker), and monitoring.

● Experience with large-scale data — high throughput, billions of rows, large parallel workloads.

● Awareness of cost optimization in compute & storage.

Good to Have

● Experience with emerging stream processors — Dagster, RisingWave, Arroyo.

● Kubernetes, Terraform, or cloud-native big-data stacks.

Mindset

● Strong ownership — takes systems from design → build → monitor.

● Self-driven, independent, and comfortable making technical decisions.

● High attention to reliability, data accuracy, and operational excellence.

● Naturally grows into broader technical responsibility as the platform scales.

Why 1DS is a great choice

● High-trust, no-politics culture — we value communication, ownership, and accountability

● Collaborative, ego-free team — building together is in our DNA

● Learning-first environment — mentorship, peer reviews, and exposure to real business impact

● Modern stack + autonomy — your voice shapes how we build

● VC-funded & scaling fast — 250+ strong, building from India for the world