Senior Data Engineer

Role Overview
We are looking for an experienced Data Engineer to design, develop, and optimize scalable data ingestion and processing pipelines. The candidate will work across batch and real-time data processing, object storage, analytical data layers, and reusable data transformation frameworks. The role requires strong hands-on engineering skills and the ability to build reliable, configurable, and maintainable data solutions.

Key Responsibilities
· Design, develop, and maintain batch and real-time data ingestion pipelines.

· Build data processing workflows using Python, SQL, Apache Spark, and PySpark.

· Implement streaming pipelines using Apache Kafka, including buffering, replay, and failure recovery.

· Develop layered data architectures covering raw, cleansed, transformed, and curated datasets.

· Build reusable components for data cleansing, joins, aggregations, validation, and transformations.

· Integrate object storage with analytical table formats and processing engines.

· Implement schema discovery, schema evolution, data quality checks, and metadata management.

· Work with structured and semi-structured formats, including Parquet, CSV, and JSON.

· Develop data decoding and transformation workflows based on provided data dictionaries, format specifications, scaling rules, and engineering units.

· Optimize Spark jobs, data partitioning, incremental processing, and analytical workloads.

· Implement logging, monitoring, error handling, and recovery mechanisms for data pipelines.

· Collaborate with backend and frontend engineers to integrate data services with application APIs and user interfaces.

· Containerize services using Docker and support development and deployment in Linux-based, on-premises environments.

· Write test cases, troubleshoot pipeline failures, and maintain technical documentation.

Required Skills

· 5–6 years of hands-on experience in data engineering and pipeline development.

· Strong proficiency in Python, SQL, Apache Spark, and PySpark.

· Practical experience with Apache Kafka and real-time or near-real-time data processing.

· Experience designing batch and streaming pipelines, including retries, checkpointing, replay, and fault handling.

· Hands-on experience with object storage such as SeaweedFS, MinIO, or S3-compatible storage.

· Experience with Apache Iceberg or comparable lakehouse table formats, including schema evolution, partitioning, and metadata management.

· Strong understanding of Parquet, CSV, JSON, schema management, and data validation.

· Experience building layered data architectures and reusable transformation frameworks.

· Experience implementing joins, aggregations, incremental processing, and data quality rules.

· Experience optimizing Spark jobs and handling large datasets efficiently.

· Familiarity with PostgreSQL or equivalent relational databases.

· Experience with REST APIs, Docker, Linux, Git, and software debugging.

· Ability to develop configurable pipelines that can be extended to new datasets and processing requirements.

Pay: ₹1,010,000.00 - ₹1,880,000.00 per year

Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka