Kafka Specialist / Senior Kafka Engineer

Job Summary:

We are looking for a skilled Kafka Specialist with strong experience in Kafka Administration, Cloudera Kafka Platform, and managing large-scale, enterprise-level messaging infrastructures. The ideal candidate will be responsible for the deployment, configuration, maintenance, and monitoring of Kafka clusters and related ecosystem components.

Key Responsibilities:

• Install, configure, upgrade, and maintain Apache Kafka and Cloudera Kafka clusters.
• Set up Kafka Connect, Kafka MirrorMaker, and Kafka REST Proxy.
• Monitor performance and fine-tune Kafka brokers, topics, partitions, producers, and consumers.
• Implement security measures including TLS encryption, Kerberos authentication, and role-based access control (RBAC).
• Develop scripts for automation using Shell, Python, or other scripting tools.
• Set up and manage Kafka Metrics, Monitoring Dashboards (Grafana, Prometheus), and log analysis tools (Splunk).
• Handle incident management and root cause analysis for Kafka-related issues.
• Collaborate with DevOps and development teams to support Kafka-based applications.
• Maintain documentation for Kafka infrastructure, SOPs, and upgrade procedures.

Required Skills & Experience:

• 5+ years of experience in Kafka Administration and Infrastructure.
• Expertise in Apache Kafka and Cloudera Kafka (CDP/CDH).
• Strong understanding of Kafka internals, producer-consumer architecture, and message delivery semantics.
• Experience with Kafka Tools like:
• Kafka Connect
• Kafka MirrorMaker
• Kafka REST Proxy
• Proficient in:
• Linux/Unix Administration
• Scripting (Shell/Python)
• Monitoring Tools (Grafana, Prometheus)
• Logging Tools (Splunk, ELK)
• Experience with SSL/TLS, SASL, Kerberos, and RBAC configuration for Kafka security.
• Familiarity with Zookeeper management and troubleshooting.
• Experience in CI/CD pipelines and DevOps practices (Jenkins, Git, etc.) is a plus.
• Understanding of Cloud environments (AWS, Azure) and containerization (Docker/Kubernetes) is a plus.