Kafka Administrator

Experience Required

10+ years overall, with 5+ years in Kafka infrastructure management and operations.
Must have successfully deployed and maintained Kafka clusters in production environments, with proven experience in securing, monitoring, and scaling Kafka for enterprise-grade data streaming.

Overview

We are seeking an experienced Kafka Administrator to lead the deployment, configuration, and operational management of Apache Kafka clusters supporting real-time data ingestion pipelines. The role involves ensuring secure, scalable, and highly available Kafka infrastructure for streaming flow records into centralized data platforms.

Role & responsibilities

• Architect and deploy Apache Kafka clusters with high availability.
• Implement Kafka MirrorMaker for cross-site replication and disaster recovery readiness.
• Integrate Kafka with upstream flow record sources using IPFIX-compatible plugins.
• Configure Kafka topics, partitions, replication, and retention policies based on data flow requirements.
• Set up TLS/SSL encryption, Kerberos authentication, and access control using Apache Ranger.
• Monitor Kafka performance using Prometheus, Grafana, or Cloudera Manager and ensure proactive alerting.
• Perform capacity planning, cluster upgrades, patching, and performance tuning.
• Ensure audit logging, compliance with enterprise security standards, and integration with SIEM tools.
• Collaborate with solution architects and Kafka developers to align infrastructure with data pipeline needs.
• Maintain operational documentation, SOPs, and support SIT/UAT and production rollout activities.

Preferred candidate profile

• Proven experience in Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.
• Strong understanding of IPFIX, nProbe Cento, and network flow data ingestion.
• Hands-on experience with Apache Spark (Structured Streaming) and modern data lake or DWH platforms.
• Familiarity with Cloudera Data Platform, HDFS, YARN, Ranger, and Knox.
• Deep knowledge of data security protocols, encryption, and governance frameworks.
• Excellent communication, documentation, and stakeholder management skills.