Kafka Developer

💼Job Title: Kafka Developer

👨 💻Job Type: Fulltime

📍Location: Pune

💼Work regime: Hybrid

🔥Keywords: Kafka, Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.

Position Overview:

What you will Have:-

Responsibilities:

Key Responsibilities

• Develop Kafka producers to ingest flow records from upstream systems such as flow record exporters (e.g., IPFIX-compatible probes).
• Build Kafka consumers to stream data into Spark Structured Streaming jobs and downstream data lakes.
• Define and manage Kafka topic schemas using Avro and Schema Registry for schema evolution.
• Implement message serialization, transformation, enrichment, and validation logic within the streaming pipeline.
• Ensure exactly once processing, checkpointing, and fault tolerance in streaming jobs.
• Integrate with downstream systems such as HDFS or Parquet-based data lakes, ensuring compatibility with ingestion standards.
• Collaborate with Kafka administrators to align topic configurations, retention policies, and security protocols.
• Participate in code reviews, unit testing, and performance tuning to ensure high-quality deliverables.
• Document pipeline architecture, data flow logic, and operational procedures for handover and support.

Required Skills & Qualifications

• Proven experience in developing Kafka producers and consumers for real-time data ingestion pipelines.
• Strong hands-on expertise in Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.
• Proficiency in Apache Spark (Structured Streaming) for real-time data transformation and enrichment.
• Solid understanding of IPFIX, NetFlow, and network flow data formats; experience integrating with nProbe Cento is a plus.
• Experience with Avro, JSON, or Protobuf for message serialization and schema evolution.
• Familiarity with Cloudera Data Platform components such as HDFS, Hive, YARN, and Knox.
• Experience integrating Kafka pipelines with data lakes or warehouses using Parquet or Delta formats.
• Strong programming skills in Scala, Java, or Python for stream processing and data engineering tasks.
• Knowledge of Kafka security protocols including TLS/SSL, Kerberos, and access control via Apache Ranger.
• Experience with monitoring and logging tools such as Prometheus, Grafana, and Splunk.
• Understanding of CI/CD pipelines, Git-based workflows, and containerization (Docker/Kubernetes)

A little about us:

Innova Solutions is a diverse and award-winning global technology services partner. We provide our clients with strategic technology, talent, and business transformation solutions, enabling them to be leaders in their field.

• Founded in 1998, headquartered in Atlanta (Duluth), Georgia.

• Employs over 50,000 professionals worldwide, with annual revenue approaching $3.0B.

• Delivers strategic technology and business transformation solutions globally.

• Operates through global delivery centers across North America, Asia, and Europe.

• Provides services for data center migration and workload development for cloud service providers.

Awardee of prestigious recognitions including:

• Women’s Choice Awards - Best Companies to Work for Women & Millennials, 2024

• Forbes, America’s Best Temporary Staffing and Best Professional Recruiting Firms, 2023

• American Best in Business, Globee Awards, Healthcare Vulnerability Technology Solutions, 2023

• Global Health & Pharma, Best Full Service Workforce Lifecycle Management Enterprise, 2023

• Received 3 SBU Leadership in Business Awards

• Stevie International Business Awards, Denials Remediation Healthcare Technology Solutions, 2023