Principal/Senior Data Engineer- Product Development

Role:

• Understand the design specifications.
• Follow the best practices for core product development.
• Data ingestion for distributed system with parallel processing, in

  Golang, CPP,

  Java for
• from cloud storage – Amazon S3/Azure Cloud Storage/Google Cloud Storage, relational database – Snowflake, Google BigQuery, PostgreSQL, Files, Kafka, data lakehouse (Iceberg)
• High-availability (HA) loading, cross-region replication
• Loading monitoring/error reporting
• Spark Connector
• Third-party management – Kafka, Kafka Connect

Requirements

• Building distributed systems applying applying parallel processing.
• Any or of the following –

  Kafka, Zookeeper

  , Spark, Stream processing.
• Experience in Kafka Connect, Kafka stream, Kafka security and customization
• Spark connectors
• Event driven architectures.
• Agile development
• CICD

Nice to have:

• gRPC protocol experience and multi-threading
• Zookeeper/ ETCD/ Consul
• Familiarity with distributed consensus algorithms (Paxos / Raft)
• Docker, Kubernetes