Senior Associate — Site Reliability Engineering (SRE)

3 years

Hyderabad

Posted:1 day ago

Job Description

Job Description – Senior Associate – SRE

Location:
Hyderabad
Work Mode: 24×7 Rotational Shifts
Experience: 3+ Years

Key Skills & Experience Required
  • Linux Administration: Strong experience with RHEL / Ubuntu, command-line operations, systemd, networking and performance troubleshooting.
  • OpenTelemetry (OTel): Hands-on experience with Java agent instrumentation, Collector pipelines, OTLP, receivers, processors and exporters.
  • http://Vector.dev: Experience with file sources, multiline parsing, VRL transforms, disk buffering and sinks.
  • Prometheus Ecosystem: Knowledge of Node Exporter, JMX Exporter, SNMP Exporter, Blackbox Exporter and PromQL.
  • Prometheus / VictoriaMetrics: Experience with scrape configurations, relabelling and remote write.
  • Log & Trace Management: Experience with VictoriaLogs (LogsQL) or OpenSearch, along with exposure to trace storage systems.
  • Apache Kafka: Understanding of topics, partitions, consumer groups, offsets, lag and retention.
  • Containerisation: Hands-on experience with Docker and working knowledge of Kubernetes.
  • Grafana: Ability to build dashboards, configure panel queries and set up alert rules.
  • Scripting & Automation: Proficiency in Bash and Python for operational automation.
  • Networking: Strong understanding of TCP/IP, routing, DNS, TLS, firewall rules and port analysis.
  • Version Control: Working knowledge of Git and configuration-as-code practices.
  • Incident Management: Experience in incident triage, escalation, root cause analysis (RCA) and documentation.
  • Willingness to work in a 24×7 rotational shift environment, including night shifts, weekends and public holidays.
Key Responsibilities
1. Ingestion Operations & Pipeline Management
  • Operate, monitor and maintain the telemetry ingestion stack across multiple data center sites, including Node Exporter, JMX Exporter, OpenTelemetry agents and collectors, Vector agents and aggregators, SNMP, Blackbox and syslog collection.
  • Manage Vector pipelines, including file sources, multiline parsing, VRL transformations, disk buffering and sink health, ensuring reliable log collection during service restarts and log rotation.
  • Manage Apache Kafka streaming infrastructure, including topic configurations, partitions, retention, replication health and consumer-group offsets.
  • Troubleshoot consumer lag, stalled consumers and ingestion failures while ensuring minimal telemetry loss.
  • Maintain consistent telemetry labels and resource attributes to enable correlation across metrics, logs and traces.
2. Onboarding, Upgrades & Data Validation
  • Deploy, configure and upgrade telemetry agents across application, Kafka, Cassandra, KeyDB, PostgreSQL, network and security hosts.
  • Validate network and security requirements, including routing, firewall rules, port connectivity, DNS resolution and TLS certificates before onboarding.
  • Perform regular health checks, capacity reviews, certificate renewals, patching and version upgrades.
  • Validate configurations and rollback procedures in lower environments before production deployment.
  • Verify telemetry completeness after every change and troubleshoot data gaps, count mismatches and cardinality issues.
3. Monitoring, Incident Management & Security
  • Monitor ingestion health through scrape success rates, ingestion volumes, consumer-group lag, buffer utilisation, queue depth and data freshness.
  • Identify and resolve silent source loss, missing telemetry and pipeline failures.
  • Build and maintain Grafana dashboards and alert rules for infrastructure and ingestion monitoring.
  • Respond to incidents within defined SLA timelines, perform root cause analysis and escalate unresolved issues through established escalation procedures.
  • Ensure secure collection endpoints by maintaining appropriate OTLP receiver bindings, read-only log mounts and preventing unauthorised access to exposed ingestion endpoints.
4. Automation & Documentation
  • Automate operational activities such as agent deployments, configuration management, verification checks, log analysis and shift reporting using Bash and Python.
  • Develop and maintain operational runbooks, Standard Operating Procedures (SOPs), onboarding guides and troubleshooting documentation.
  • Maintain incident records, RCA reports and shift handover documentation.
  • Identify opportunities to improve operational efficiency and reduce manual intervention.
5. 24×7 Support & SLA Management
  • Provide continuous operational support across morning, evening and night shifts, including weekends and public holidays.
  • Ensure structured shift handovers covering open incidents, ingestion health, suppressed alerts and ongoing activities. All handovers must be documented.
  • Follow defined incident response and escalation timelines:
  • P1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, with further escalation up to the Incident Commander at 30 minutes.
  • P2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.
  • Participate in structured shift overlap and backfill arrangements to ensure uninterrupted operational coverage during planned and unplanned absences.
Additional Requirements
  • Strong troubleshooting and analytical skills.
  • Ability to work independently in a production support environment.
  • Good communication, documentation and incident coordination skills.
  • Comfortable working in a critical 24×7 production environment with strict SLA commitments.

Skills Required

Site Reliability Engineering Linux Administration Networking Troubleshooting Java Apache Kafka Docker Kubernetes Automation Python Tcp/ip

Mock Interview

Practice Video Interview with JobPe AI

Start Java Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Java Skills

Practice Java coding challenges to boost your skills

Start Practicing Java Now
WITS INNOVATION LAB logo
WITS INNOVATION LAB

Technology/Innovation

Johannesburg