Freelance Job Opportunity – Kafka/Streaming Data Engineer

Company Description

ThreatXIntel is a startup cyber security company dedicated to protecting businesses and organizations from cyber threats. We provide services such as cloud security, web and mobile security testing, and DevSecOps. Our goal is to offer customized, affordable solutions to meet the specific needs of our clients. By continuously monitoring and testing digital environments, we identify vulnerabilities before they can be exploited. At ThreatXIntel, our mission is to deliver exceptional cyber security services that allow our clients to focus on growing their businesses.

experienced Kafka/Streaming Data Engineer (Freelance/Contract)

hands-on expertise in Apache Kafka cluster management, automation, monitoring, and performance tuning

Key Responsibilities

• Kafka Administration & Optimization

• Configure, manage, and tune multi-node

  Kafka clusters

  for high throughput, low latency, and fault tolerance.
• Define and optimize

  partitions, replication, retention policies, and consumer groups

  .
• Ensure stability of event-driven pipelines across on-prem and AWS cloud environments.

• Automation & Infrastructure as Code

• Develop and maintain

  Ansible playbooks

  for Kafka cluster provisioning, deployment, scaling, and upgrades.
• Automate repetitive operational tasks (broker restarts, topic creation, retention updates, etc.).
• Build reliable solutions for

  data replication between on-premise and AWS cloud

  .

• Monitoring & Observability

• Implement

  Prometheus and Grafana dashboards

  for Kafka brokers, producers, and consumers.
• Track and alert on

  lag, throughput, latency, and error rates

  .
• Perform

  load testing and failure simulations

  to validate production readiness.

• Streaming Pipeline Support

• Monitor and troubleshoot

  Ab Initio CDC jobs, Kafka topics, and Apache Flink consumers

  .
• Ensure data integrity and timely delivery into downstream databases.
• Support

  UAT and production go-live

  with synthetic volume tests.

• Troubleshooting & Performance Tuning

• Diagnose and resolve issues related to

  broker performance, consumer lag, data loss, or scaling bottlenecks

  .
• Propose

  short-term fixes

  and implement

  long-term architectural improvements

  .

Required Skills & Experience

• Kafka Expertise:

• Strong hands-on experience with

  Kafka cluster administration in production-grade environments

  .
• Deep understanding of

  broker tuning, partitioning strategies, consumer offsets, replication, and retention policies

  .

• Automation & Scripting:

• Proficiency with

  Ansible

  for automated deployments and configuration management.
• Strong scripting skills in

  Python, Bash, or Shell

  .

• Monitoring & Alerting:

• Hands-on experience with

  Prometheus and Grafana

  setup, tuning, and alert configuration.

• Streaming & Data Engineering:

• Knowledge of

  Apache Flink

  or equivalent real-time stream-processing frameworks.
• Familiarity with

  CDC tools (Ab Initio CDC preferred)

  .

• Databases:

• Good understanding of

  PostgreSQL

  and integration patterns with Kafka.

• Production Experience:

• Worked with

  large-scale, high-throughput data pipelines (multi-billion row datasets)

  .
• Experience troubleshooting

  real-time ingestion systems under heavy load

  .

• Soft Skills:

• Strong problem-solving skills with a

  “production-first” mindset

  .
• Excellent communication for cross-team collaboration.