Senior Big Data Infrastructure Engineer

Purpose

Key Responsibilities

• Design and automate on-premise deployments/upgrades for Kafka, Spark, and related big data tools using tools like Kubernetes or Salt, targeting zero-downtime operations.
• Configure and optimize distributed storage systems (e.g., MinIO or equivalent on-prem solutions) for large-scale log data, including capacity planning, monitoring, and data integrity checks.
• Implement monitoring and alerting for infrastructure health (e.g., using Prometheus/Grafana), with auto-remediation for issues like CVEs or performance bottlenecks.
• Enforce on-prem security measures: RBAC, system hardening, SSL/TLS, and compliance auditing to mitigate risks in high-volume data environments.
• Propose and document infrastructure solutions (e.g., "Scaling on-prem Kafka clusters for 100x log volume without data loss") and collaborate with development teams on integration points like observability.
• Handle big data volume management, including automation for Spark job scheduling and resource allocation in on-prem clusters.

Technical Requirements

• Core Skills:

• 5+ years managing on-premise Linux/Unix systems for mission-critical, high-volume big data operations (e.g., Hadoop/Spark clusters).
• 3+ years with Kubernetes, CI/CD pipelines, IaC (Salt/Terraform/Ansible), and monitoring tools (Prometheus/Grafana).
• Deep expertise in Kafka for streaming, MinIO or similar on-prem storage, and security tools (e.g., OpenVAS, auditd).

• Security Focus

  : Rapid CVE patching (within 72 hours), anomaly detection, and on-prem compliance standards.

• Flexibility

  : Experience with GCP/AWS is a plus for understanding core concepts and tools (e.g., Kinesis or S3 equivalents).

Soft Skills

• Proactive in identifying and solving infrastructure risks (e.g., scalability, security for big data logs).
• Strong documentation of configurations, solutions, and rationales for team use.
• Collaborative mindset to guide developers on infra-related integrations without overlapping into app development.

Prefered Industry Experience

• Priority Sectors:

  Banking, Tech, Retail, Healthcare (with on-prem data centers).

• Candidate Profile:

  Seniors from enterprises with on-premise or hybrid big data infrastructures handling massive log volumes (e.g., 1M+ events/day).