Senior Observability Engineer

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

Senior Observability Engineer

Location:

Eli Lilly, Hyderabad

Position Type:

Full-Time

Level:

P2–P3

At Lilly, we believe in the talent of our workforce. One of the best ways to utilize and develop that talent is to use our existing workforce to fill new and/or open positions. If you are looking for a new position within Lilly, you can view and apply to open roles posted in the internal job posting system. You must meet the minimum qualifications outlined in the job description and have/obtain work authorization in the country the position is located in order to be considered. When applying internally for a position, your current supervisor will receive notification that you have applied to the position. We encourage employees to discuss the opportunity with their supervisor prior to applying.

Note: Roles are posted at the lowest level of a band; however, employees should search across all levels of the band to identify all opportunities. Employees hired on banded positions (e.g., P1–P3, R1–R2, B1–B3, etc.) transfer at their current level, despite the level indicated on the job posting.

Observability Engineer – Tech@Lilly

We are seeking a skilled and motivated Senior Observability Engineer to join our growing team. This role is critical to enhancing operational visibility, enabling proactive issue detection, and improving system reliability across Lilly’s digital ecosystem. You will work with cutting-edge observability tools such as Splunk Cloud, Splunk IT Service Intelligence (ITSI), and OpenTelemetry, collaborating with cross-functional teams to build robust, scalable monitoring solutions.

Key Responsibilities

Data Ingestion & Integration

• Design, implement, and manage data ingestion pipelines from diverse sources into Splunk Cloud.
• Ensure data quality, efficiency, and compliance across ingestion workflows.
• Implement and manage Open Telemetry for standardized collection of traces, metrics, and logs.

Splunk ITSI Configuration

• Develop and maintain ITSI components including:
  • Service Tree Design: Map IT infrastructure and applications into logical service trees reflecting business criticality.
  • KPI Creation: Define and refine KPIs to measure service health and performance.
  • Glass Table Development: Build intuitive dashboards for real-time operational insights.
  • Anomaly Detection: Configure machine learning-based anomaly detection and intelligent thresholds.

Performance Optimization

• Tune Splunk Cloud and ITSI for optimal performance, including search execution and dashboard responsiveness.
• Implement Logs as a Code (LaaC) practices for managing logging configurations and parsing rules in a version-controlled, automated manner.

Troubleshooting & Support

• Provide expert-level support for Splunk and ITSI-related issues, including data discrepancies and performance bottlenecks.
• Collaborate with SREs, developers, and operations teams to translate observability needs into actionable solutions.

Collaboration & Documentation

• Work closely with application and infrastructure teams to align observability strategies with business goals.
• Maintain comprehensive documentation and promote observability best practices across the organization.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.
• 3–6 years of hands-on experience with Splunk Cloud and/or Splunk ITSI in enterprise environments.
• Strong proficiency in Splunk SPL, including advanced queries, macros, and data model acceleration.
• Experience with Open Telemetry instrumentation and observability pipelines.
• Familiarity with scripting languages (e.g., Python, Bash) for automation and API integrations.
• Understanding of cloud platforms (AWS, Azure, GCP) and container technologies (Docker, Kubernetes) is a plus.

Preferred Certifications

• Splunk Core Certified User or Administrator
• Splunk ITSI Certified Admin
• Splunk Enterprise Certified Architect (bonus)

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly