Lead DevOps Engineer

Role Overview

We are seeking a DevOps Lead to architect, lead, and scale the infrastructure and automation ecosystem for a next-generation data Lakehouse product. The ideal candidate will combine deep technical expertise in infrastructure automation, CI/CD, observability, and multi-cloud operations with leadership skills to guide a DevOps engineering team.

Key Responsibilities:

Leadership & Strategy

• Lead a team of DevOps/SRE engineers; mentor, review, and set best practices.
• Define a DevOps roadmap aligned with product requirements (scalability, cost-optimization, security).
• Own infra observability, reliability SLAs, and incident response practices.

Infrastructure & Automation

• Architect and manage hybrid deployments across on-prem YARN clusters, AWS (S3, EMR, EKS
• Build and maintain infrastructure-as-code (IaC) with Terraform/Helm/Ansible.
• Standardize CI/CD pipelines (Jenkins/GitHub Actions) for Java/Scala (Spring Boot, Spark) microservices.

Platform Operations

• Ensure high availability of Spark, Trino, Hive Metastore, and ADS services.
• Implement monitoring/alerting with Prometheus, Grafana, ELK, and custom dashboards.
• Optimize storage costs with tiering (S3 Standard/IA/Glacier, Ceph/HDFS).

Security & Compliance

• Integrate secrets management (Vault, KMS/HSM).
• Enforce compliance frameworks (GDPR, HIPAA, PCI-DSS) via DevSecOps practices.
• Govern IAM roles, TLS certs, network whitelisting/blacklisting policies.
• Collaboration
• Partner with Architects and Developers on release processes and production readiness.
• Drive blameless postmortems and continuous improvement.

Collaboration

• Partner with Architects and Developers on release processes and production readiness.
• Drive blameless postmortems and continuous improvement.

Required Skills

• 68 years of DevOps/SRE experience, with 1-3 years in leadership.
• Strong in Kubernetes, YARN, Spark, Iceberg, Trino/Hive operations.
• Deep expertise in Terraform, Ansible, Helm, GitHub Actions/Jenkins.
• Hands-on with AWS & Azure services (IAM, S3, Blob, ADLS, Service Bus, Event Hubs).
• Experience with monitoring/observability stack (Prometheus, Grafana, ELK).
• Strong knowledge of networking, firewalls, VPNs, DNS, load balancers (ALB/NLB).
• Excellent leadership, problem-solving, and communication skills.