OTTObserve Academy

Learning Objectives

Distinguish the three pillars — metrics, logs, and traces — by their cost, cardinality, and question they best answer.
Map a single OTT play request across the Mobile -> Gateway -> Auth -> Playback -> DRM -> Postgres chain.
Explain why a video-start-failure spike is a metrics question but the root cause is a traces question.
Identify which pillar you would reach for first during a live-match buffering incident.

Key Concepts

Metrics: cheap, aggregate, low-cardinality

Metrics are numeric time series (counters, gauges, histograms) sampled at a fixed interval. They answer 'how much / how many / how fast' across a whole fleet and are the cheapest pillar per data point. In OTT you track video_start_failures_total, rebuffer_ratio, and playback_latency_seconds. Their weakness is that a single label with unbounded values (device_ip, user_uuid) explodes cardinality and cost.

Logs: high-detail, per-event, expensive at scale

Logs capture discrete events with rich context (a DRM license denial, a 504 from the playback service). They answer 'what exactly happened for this request'. Centralized in EFK/Loki, they are indexed for search. They are verbose and costly, so at 250K RPS you sample and structure them rather than logging every line.

Traces: causal, cross-service, the 'why is it slow' pillar

A trace stitches spans from every service a request touched into one causal tree sharing a W3C trace ID. Traces answer 'where did the latency go' and 'which downstream call failed'. For OTT, a trace shows a play request fanning out to Auth, then Playback, then a DRM license round-trip and a Postgres entitlement lookup.

The OTT playback flow

A play tap travels Mobile -> API Gateway -> Auth (token + entitlement) -> Playback (manifest + CDN URL) -> DRM (license) -> Postgres (entitlement/session). Latency or errors at any hop degrade video-start. Observability instruments every hop so you can localize failure to a single service instead of guessing.

Insight

Start with metrics to detect and quantify, pivot to traces to localize, then drop into logs to read the exact failing event. This detect -> localize -> read loop is the core SRE triage motion.

Warning

Putting a high-cardinality label (user_uuid, device_ip, session_id) on a Prometheus metric turns one series into millions. This is the single most common way OTT teams blow up their metrics bill.

Info

The three pillars are complementary, not competing. Mature OTT platforms correlate all three via a shared trace ID injected into logs and exemplars on metrics.

Production Configs & Runbooks

Prometheus recording rule: video-start failure ratioyaml

groups:
  - name: ott-playback.rules
    interval: 15s
    rules:
      - record: playback:video_start_failure_ratio:5m
        expr: |
          sum(rate(video_start_failures_total[5m]))
            /
          sum(rate(video_start_attempts_total[5m]))
      - record: playback:rebuffer_ratio:5m
        expr: |
          sum(rate(rebuffer_events_total[5m]))
            /
          sum(rate(playback_sessions_total[5m]))

OTel span attributes for a playback request (Go)go

ctx, span := tracer.Start(ctx, "playback.resolve_manifest")
defer span.End()

span.SetAttributes(
    attribute.String("ott.content_id", contentID),
    attribute.String("ott.cdn.pop", pop),
    attribute.String("ott.drm.scheme", "widevine"),
    // NOTE: never attach user_uuid / device_ip as a *metric* label,
    // but it is safe (and useful) as a *span* attribute.
    attribute.String("ott.session_id", sessionID),
)

license, err := drm.Fetch(ctx, contentID)
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, "drm license fetch failed")
}

Structured playback log line (JSON) correlated by trace_idjson

{
  "level": "error",
  "service": "playback",
  "msg": "video start failed: drm license denied",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "ott.content_id": "match-ind-vs-aus-live",
  "ott.cdn.pop": "mum-1",
  "ott.drm.scheme": "widevine",
  "http.status_code": 403
}

Pillars of Observability & the OTT Playback Flow