OTTObserve Academy

Learning Objectives

Contrast head sampling (decide at trace start) with tail sampling (decide after the trace completes).
Explain why tail sampling can guarantee capture of all error and slow traces.
Describe the memory/latency cost of buffering traces for a tail decision at the gateway.
Choose a sampling strategy for a live match given a fixed trace budget.

Key Concepts

Head sampling

Head sampling decides keep/drop at the first span using only information available at trace start (a probability, maybe the route). It is cheap and stateless but blind — it may drop the exact error trace you needed because it could not know the trace would fail.

Tail sampling

Tail sampling buffers all spans of a trace and decides after completion, so policies can key on outcome: keep every ERROR, keep every trace over a latency threshold, and probabilistically sample the healthy baseline. This guarantees you keep the interesting traces.

Cost of the tail decision

Buffering whole traces until decision_wait costs memory and adds decision latency at the gateway, and requires all spans of a trace to reach the same Collector (span-aware routing). This is why tail sampling lives on the gateway, not the agent.

Choosing a strategy under budget

For a marquee match: tail-sample so 100% of errors and slow traces survive while the healthy baseline is sampled at a low percentage, keeping total volume within budget without losing diagnostic value.

Insight

Head sampling optimizes for cost and simplicity; tail sampling optimizes for keeping the traces you will actually want at 2am. For incident-grade OTT, tail sampling wins.

Warning

Tail sampling requires span-aware routing so ALL spans of one trace land on the same gateway Collector. Round-robin load balancing breaks the decision.

Info

decision_wait must exceed your longest expected trace, or slow traces get a decision before they finish and are misclassified.

Production Configs & Runbooks

Tail sampling policies (keep errors + slow, sample baseline)yaml

processors:
  tail_sampling:
    decision_wait: 12s          # must exceed longest expected trace
    num_traces: 200000
    expected_new_traces_per_sec: 20000
    policies:
      - name: keep-all-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow-traces
        type: latency
        latency:
          threshold_ms: 1200
      - name: baseline-probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

Head (probabilistic) sampler for comparison (SDK-side)go

// Head sampling: decision made at trace start, before any outcome is known.
tp := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(
        sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.05), // keep ~5% blindly
        ),
    ),
)
// Limitation: a failing DRM request in the dropped 95% is lost forever,
// because head sampling cannot know the trace will error.

Span-aware load balancing so all spans hit one gatewayyaml

exporters:
  loadbalancing:
    routing_key: traceID     # all spans of a trace route to the same backend
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-gateway.observability.svc

Production configs and runbooks are a Pro feature.

Tail vs Head Sampling