Course Overview
This course enables participants to build, configure, and troubleshoot a full observability pipeline using the Splunk Distribution of the OpenTelemetry Collector. Learners will gain experience with metrics, logs, and traces, exploring how to enrich, transform, and analyze telemetry data flowing through the collector into Splunk Observability Cloud.
Who should attend
- This course is most useful for SRE, DevOps Engineers, Platform Engineers and Observability Monitoring Specialists.
- The course can be relevant for Application developers too.
Prerequisites
- Basic Linux terminal proficiency
- Familiarity with Docker, Docker Compose
- Working knowledge of YAML
- Awareness of telemetry concepts (metrics, logs, traces)
- Introductory experience with Splunk Observability Cloud or similar observability backends
Outline: Full Observability Pipeline with Splunk OpenTelemetry (FOPSOT)
Module 1: Splunk Observability Cloud
- Differentiate between monitoring and observability
- Describe the telemetry data for observability
- Describe Splunk Observability Cloud architecture
- Navigate Splunk Observability cloud
- Use built-in content to Interpret system health and activity
Module 2: Introduction to the OpenTelemetry Collector
- Describe OpenTelemetry and available distributions
- Describe OpenTelemetry Collector packaging options, deployment models and ingestion modes
- Visualize the flow of telemetry data through the Collector pipeline, from ingestion to export
- Deploy and configure the OpenTelemetry Collector using guided steps
- Use best practices and advanced configuration techniques
Module 3 – Traces Pipeline
- Use the `otlphttp` exporter to send traces to Splunk Observability Cloud
- Compare auto and manual instrumentation approaches
- Auto-instrument a Node.js Express application using the Splunk
- OpenTelemetry JavaScript agent
- Enrich traces with additional metadata using processors
- Explore trace data and errors using the APM Service Map
Module 4 – Metrics Pipeline
- Define processors to rename, aggregate, and scale metrics
- Configure the signalfx exporter
- Explain the importance of metric normalization
- Apply best practices when editing the otelcol-config.yml file
Module 5 – Logs Pipeline
- Describe log ingestion into Splunk via splunk_hec and how logs become viewable in Splunk Observability Cloud.
- Use Log Observer to search and explore logs
- Filter noisy logs with the filter processor to reduce telemetry volume and control data costs.
- Redact, or modify log data to meet privacy and compliance needs
Module 6 – Putting it all Together
- Create custom charts, dashboards, and alerts for telemetry data
- Use telemetry signals and related content to find issues and troubleshoot root cause of problems