Kafka Developer

Overview:

We are looking for a Kafka Developer to design and implement real-time data ingestion

pipelines using Apache Kafka. The role involves integrating with upstream flow record sources,

transforming and validating data, and streaming it into a centralized data lake for analytics and

operational intelligence.

Required Skills & Qualifications:

• Proven experience in developing Kafka producers and consumers for real-time data

ingestion pipelines.

• Strong hands-on expertise in Apache Kafka, Kafka Connect, Kafka Streams, and

Schema Registry.

• Proficiency in Apache Spark (Structured Streaming) for real-time data transformation

and enrichment.

• Solid understanding of IPFIX, NetFlow, and network flow data formats; experience

integrating with nProbe Cento is a plus.

• Experience with Avro, JSON, or Protobuf for message serialization and schema

evolution.

• Familiarity with Cloudera Data Platform components such as HDFS, Hive, YARN, and

Knox.

• Experience integrating Kafka pipelines with data lakes or warehouses using Parquet or

Delta formats.

• Strong programming skills in Scala, Java, or Python for stream processing and data

engineering tasks.

• Knowledge of Kafka security protocols including TLS/SSL, Kerberos, and access control

via Apache Ranger.

• Experience with monitoring and logging tools such as Prometheus, Grafana, and Splunk.
• Understanding of CI/CD pipelines, Git-based workflows, and containerization

(Docker/Kubernetes)