Splunk Developer – Machine Learning & Observability Expert

Position Overview

We are seeking an experienced Splunk Developer to join Enterprise Monitoring team. The ideal candidate will have 6-8 years of hands-on experience with Splunk, including search optimization, machine learning capabilities, and deep technical expertise in log analysis and monitoring solutions.

Key Responsibilities

1. Machine Learning & AIOps

• Design and implement ML-based monitoring solutions

  using

  Splunk’s ML Toolkit (MLTK)

  .

• Build predictive and anomaly detection models

  for infrastructure metrics (CPU, memory, latency, etc.).

• Develop custom ML use cases

  —such as log clustering, failure prediction, and capacity forecasting.

• Optimize real-time analytics

  for large-scale datasets (millions of events/sec).

2. Splunk Observability & Unified Dashboards

• Implement end-to-end observability

  using

  Splunk Observability Cloud (APM, RUM, Log Observer, Infrastructure Monitoring)

  .

• Design unified dashboards

  that consolidate

  metrics, traces, and logs

  across hybrid cloud (AWS/Azure/GCP) and on-prem systems.

• Correlate ML insights with observability data

  to automate root cause analysis (RCA).

• Integrate with OpenTelemetry, Prometheus, and distributed tracing

  for full-stack visibility.

3. Search Optimization & Scalability

• Reduce search overhead

  by optimizing

  SPL queries

  and data model acceleration.

• Implement summary indexing

  and

  data sampling

  for high-volume environments.

4. Automation & Advanced Analytics

• Python scripting

  for

  custom ML pipelines, API integrations, and automation

  .

• Leverage Splunk’s REST API

  for dynamic dashboarding and alerting.

• Splunk App Development & Integrations

  : Build custom apps and integrate Splunk with

  third-party tools (ITSM, CI/CD, Cloud platforms)

  .

Mandatory Skills

6-8 years of Splunk development

Hands-on Splunk Observability Cloud (SignalFX/APM/IM/Log Observer)

Experience building unified dashboards for infrastructure (servers, Kubernetes, cloud, network)

Strong Python for ML (Pandas, Scikit-learn, TensorFlow/PyTorch is a plus)

Search optimization at scale (data models, accelerated reports, summary indexing)

✅ Familiarity with DevOps practices (CI/CD pipelines, Terraform, Ansible).

✅ Experience with OpenTelemetry, Prometheus, and distributed tracing.

✅ Knowledge of IT operations, SRE (Site Reliability Engineering), and incident management.

Splunk certifications (MLTK, Observability, Core Certified Power User)

Knowledge of MLOps pipelines

Experience with OpenTelemetry, Prometheus, and Grafana integrations

Why This Role?

not just a Splunk admin role

Build AI-driven monitoring

Turn observability data into actionable insights

Optimize performance

built ML models in Splunk and designed observability solutions for complex environments,