Job Description – Senior Associate – SRE
Location: Hyderabad
Work Mode: 24×7 Rotational Shifts
Experience: 3+ Years
Key Skills & Experience Required
-
Linux Administration: Strong experience with RHEL / Ubuntu, command-line operations, systemd, networking and performance troubleshooting.
-
OpenTelemetry (OTel): Hands-on experience with Java agent instrumentation, Collector pipelines, OTLP, receivers, processors and exporters.
-
http://Vector.dev: Experience with file sources, multiline parsing, VRL transforms, disk buffering and sinks.
-
Prometheus Ecosystem: Knowledge of Node Exporter, JMX Exporter, SNMP Exporter, Blackbox Exporter and PromQL.
-
Prometheus / VictoriaMetrics: Experience with scrape configurations, relabelling and remote write.
-
Log & Trace Management: Experience with VictoriaLogs (LogsQL) or OpenSearch, along with exposure to trace storage systems.
-
Apache Kafka: Understanding of topics, partitions, consumer groups, offsets, lag and retention.
-
Containerisation: Hands-on experience with Docker and working knowledge of Kubernetes.
-
Grafana: Ability to build dashboards, configure panel queries and set up alert rules.
-
Scripting & Automation: Proficiency in Bash and Python for operational automation.
-
Networking: Strong understanding of TCP/IP, routing, DNS, TLS, firewall rules and port analysis.
-
Version Control: Working knowledge of Git and configuration-as-code practices.
-
Incident Management: Experience in incident triage, escalation, root cause analysis (RCA) and documentation.
-
Willingness to work in a 24×7 rotational shift environment, including night shifts, weekends and public holidays.
Key Responsibilities
1. Ingestion Operations & Pipeline Management
-
Operate, monitor and maintain the telemetry ingestion stack across multiple data center sites, including Node Exporter, JMX Exporter, OpenTelemetry agents and collectors, Vector agents and aggregators, SNMP, Blackbox and syslog collection.
-
Manage Vector pipelines, including file sources, multiline parsing, VRL transformations, disk buffering and sink health, ensuring reliable log collection during service restarts and log rotation.
-
Manage Apache Kafka streaming infrastructure, including topic configurations, partitions, retention, replication health and consumer-group offsets.
-
Troubleshoot consumer lag, stalled consumers and ingestion failures while ensuring minimal telemetry loss.
-
Maintain consistent telemetry labels and resource attributes to enable correlation across metrics, logs and traces.
2. Onboarding, Upgrades & Data Validation
-
Deploy, configure and upgrade telemetry agents across application, Kafka, Cassandra, KeyDB, PostgreSQL, network and security hosts.
-
Validate network and security requirements, including routing, firewall rules, port connectivity, DNS resolution and TLS certificates before onboarding.
-
Perform regular health checks, capacity reviews, certificate renewals, patching and version upgrades.
-
Validate configurations and rollback procedures in lower environments before production deployment.
-
Verify telemetry completeness after every change and troubleshoot data gaps, count mismatches and cardinality issues.
3. Monitoring, Incident Management & Security
-
Monitor ingestion health through scrape success rates, ingestion volumes, consumer-group lag, buffer utilisation, queue depth and data freshness.
-
Identify and resolve silent source loss, missing telemetry and pipeline failures.
-
Build and maintain Grafana dashboards and alert rules for infrastructure and ingestion monitoring.
-
Respond to incidents within defined SLA timelines, perform root cause analysis and escalate unresolved issues through established escalation procedures.
-
Ensure secure collection endpoints by maintaining appropriate OTLP receiver bindings, read-only log mounts and preventing unauthorised access to exposed ingestion endpoints.
4. Automation & Documentation
-
Automate operational activities such as agent deployments, configuration management, verification checks, log analysis and shift reporting using Bash and Python.
-
Develop and maintain operational runbooks, Standard Operating Procedures (SOPs), onboarding guides and troubleshooting documentation.
-
Maintain incident records, RCA reports and shift handover documentation.
-
Identify opportunities to improve operational efficiency and reduce manual intervention.
5. 24×7 Support & SLA Management
-
Provide continuous operational support across morning, evening and night shifts, including weekends and public holidays.
-
Ensure structured shift handovers covering open incidents, ingestion health, suppressed alerts and ongoing activities. All handovers must be documented.
-
Follow defined incident response and escalation timelines:
-
P1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, with further escalation up to the Incident Commander at 30 minutes.
-
P2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.
-
Participate in structured shift overlap and backfill arrangements to ensure uninterrupted operational coverage during planned and unplanned absences.
Additional Requirements
-
Strong troubleshooting and analytical skills.
-
Ability to work independently in a production support environment.
-
Good communication, documentation and incident coordination skills.
-
Comfortable working in a critical 24×7 production environment with strict SLA commitments.