OTTObserve Academy

Learning Objectives

Trace how ingress/egress buffer pressure develops under a live-match surge.
Diagnose Redis connection-pool exhaustion (COBL) as the true root cause behind cascading playback failures.
Connect client output buffer limits, eviction, and latency to observable metrics.
Choose mitigations that address the bottleneck rather than the symptom.

Key Concepts

Ingress/egress buffers under surge

During the toss-to-first-ball surge, connection and output buffers fill faster than they drain. Egress buffer growth and rising queue depth are early tells that a downstream dependency (often Redis) has become the constraint.

Redis connection-pool exhaustion

When the app's Redis pool is exhausted, requests queue waiting for a connection; latency climbs and eventually times out, cascading into playback/session failures. The symptom is video-start failures, but the root cause is pool saturation, not the playback service.

Client Output Buffer Limits (COBL)

Redis enforces client-output-buffer-limits; a slow or overloaded consumer whose output buffer exceeds the limit is disconnected, dropping pub/sub or replica streams. Under surge this manifests as intermittent session-state loss.

Fix the bottleneck, not the symptom

Restarting the playback pods (the symptom) does nothing. The right moves target the constraint: raise/scale the Redis connection pool, add read replicas, shed load, and correct output-buffer limits.

Warning

Video-start failures point at playback, but the true root cause is frequently Redis connection-pool exhaustion upstream. Restarting playback pods treats the symptom and the incident recurs immediately.

Insight

Watch pool wait time and blocked-client count, not just Redis CPU. Exhaustion shows up as clients WAITING for a connection long before CPU saturates.

Info

The cardinality-shock and cricket-buffer-storm War Rooms exercise exactly this diagnosis: correlate rising egress buffers, Redis pool wait, and playback failures to the real bottleneck.

Production Configs & Runbooks

Redis client output buffer limits (COBL) hardeningproperties

# redis.conf — client output buffer limits (COBL)
# format: <class> <hard limit> <soft limit> <soft seconds>
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 512mb 128mb 60
client-output-buffer-limit pubsub 64mb 16mb 60
maxclients 20000
# surface exhaustion via INFO clients: blocked_clients / connected_clients

Application connection pool sized for match peak (Node/ioredis)javascript

// Pool sized for peak concurrency; expose wait time as a metric.
const pool = new Redis.Cluster(nodes, {
  // bound the pool so exhaustion is observable, not silently unbounded
  maxRedirections: 16,
  scaleReads: "slave",         // offload reads to replicas under surge
  redisOptions: {
    connectTimeout: 500,
    maxRetriesPerRequest: 2,
    enableOfflineQueue: false, // fail fast instead of queueing forever
  },
});

// Emit pool wait time so triage sees exhaustion before timeouts cascade
poolAcquireHistogram.observe(waitMs / 1000);

Alert on Redis pool exhaustion (root cause, not symptom)yaml

groups:
  - name: redis-bottleneck.rules
    rules:
      - alert: RedisConnectionPoolExhaustion
        expr: |
          max by (service) (redis_blocked_clients)
            > 0
          and
          histogram_quantile(0.99, sum by (le, service) (
            rate(redis_pool_acquire_wait_seconds_bucket[1m])
          )) > 0.25
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "Redis pool exhaustion driving playback failures"
          description: "Clients blocked waiting for connections; p99 acquire wait > 250ms. Scale pool / add replicas — do NOT restart playback."

Production configs and runbooks are a Pro feature.

Ingress/Egress, Buffer Overflows & Redis Bottlenecks (COBL)