Data Platform Engineer

Data Platform Engineer

Key Responsibilities:

1. Data Ingestion (ETL Engine):

• Design and maintain data pipelines to ingest from:
• File systems: CSV, Excel, PDF, binary formats
• Databases: Using JDBC connectors (PostgreSQL, MySQL, etc.)
• APIs: REST, XML, GraphQL endpoints
• Implement and optimize:

• Airflow

  for scheduling and orchestration

• Apache NiFi

  for drag-and-drop pipeline development

• Kafka / Redis Streams

  for real-time or event-based ingestion
• Develop

  custom Python connectors

  for air-gapped environments
• Handle binary data using

  PyPDF2

  ,

  protobuf

  ,

  OpenCV

  ,

  Tesseract

  , etc.
• Ensure secure storage of raw data in

  MinIO

  ,

  GlusterFS

  , or other vaults

2. Transformation Layer:

• Implement SQL/code-based transformation using:

• dbt-core

  for modular SQL pipelines

• Dask

  or

  Pandas

  for mid-size data processing

• Apache Spark

  for large-scale, distributed ETL
• Integrate

  Great Expectations

  or other frameworks for data quality validation (optional in on-prem)
• Optimize data pipelines for latency, memory, and parallelism
  

3. Data Warehouse (On-Prem):

• Deploy and manage on-prem OLAP/RDBMS options including:

• ClickHouse

  for real-time analytics

• Apache Druid

  for event-driven dashboards

• PostgreSQL

  ,

  Greenplum

  , and

  DuckDB

  for varied OLAP/OLTP use cases
• Architect

  multi-schema / multi-tenant

  isolation strategies
• Maintain warehouse performance and data consistency across layers
  

4. BI Dashboards:

• Develop and configure per-tenant dashboards using:

• Metabase

  (preferred for RBAC + multi-tenant)

• Apache Superset

  or

  Redash

  for custom exploration

• Grafana

  for technical metrics
• Embed dashboards into customer portals
• Configure PDF/Email-based scheduled reporting
• Work with stakeholders to define marketing, operations, and executive KPIs
  

Required Skills & Qualifications:

• 5+ years of hands-on experience with

  ETL tools

  ,

  data transformation

  , and

  BI platforms

• Advanced Python skills for custom ingestion and transformation logic
• Strong understanding of

  SQL

  ,

  data modeling

  , and

  query optimization

• Experience with

  Apache NiFi

  ,

  Airflow

  ,

  Kafka

  , or

  Redis Streams

• Familiarity with at least two:

  ClickHouse

  ,

  Druid

  ,

  PostgreSQL

  ,

  Greenplum

  ,

  DuckDB

• Experience building

  multi-tenant data platforms

• Comfort working in

  air-gapped / on-prem environments

• Strong understanding of

  security, RBAC

  , and data governance practices

Nice-to-Have Skills:

• Experience in regulated industries (BFSI, Telecom, government)
• Knowledge of

  containerization

  (Docker/Podman) and orchestration (K8s/OpenShift)
• Exposure to

  data quality and validation frameworks

  (e.g., Great Expectations)
• Experience with embedding BI tools in web apps (React, Django, etc.)
  

What We Offer:

• Opportunity to build a cutting-edge, open-source-first data platform for real-time insights
• Collaborative team environment focused on secure and scalable data systems
• Competitive salary and growth opportunities