OTTObserve Academy

Learning Objectives

Design an observability pipeline that survives a 250K-500K RPS live cricket match without collapsing under its own telemetry.
Apply a streaming buffer (Kafka/Kinesis) between producers and the metrics/trace backends.
Balance sampling, aggregation, and retention to keep the telemetry bill sane at peak.
Reason about shard/partition lag as a leading indicator of ingestion backpressure.

Key Concepts

Telemetry is a workload too

At 250K-500K RPS the telemetry itself becomes a high-throughput system. Naive per-request metrics and 100% tracing will overwhelm ingestion before the app does. Observability must be capacity-planned like any tier-0 service.

Streaming buffer (Kafka / Kinesis)

Placing Kafka or Kinesis between telemetry producers and backends absorbs bursts, decouples producer spikes from backend capacity, and lets you replay on backend outage. Partition/shard count sets your parallelism ceiling.

Shard/partition lag as backpressure signal

Rising consumer lag on the telemetry stream is the earliest sign the backend cannot keep up. It precedes dropped spans and gaps in metrics, so it is the leading indicator to alert on during a match.

Sampling + aggregation + retention

Keep 100% of errors and slow traces (tail sampling), probabilistically sample the baseline, pre-aggregate high-volume metrics with recording rules, and tier retention (hot local, cold object storage) to control cost at peak.

Warning

If your telemetry pipeline has no buffer, a backend slowdown during the toss-to-first-ball surge applies backpressure straight to your application, so your observability outage becomes an application outage.

Insight

Alert on consumer/shard LAG, not just on drops. By the time you are dropping spans, you have already lost the data you needed to debug the surge.

Info

Peak concurrency for a marquee match is spiky and predictable: pre-scale the gateway Collector fleet and stream partitions ahead of the scheduled start.

Production Configs & Runbooks

Kinesis-buffered telemetry ingestion (Collector exporter)yaml

exporters:
  awskinesis:
    aws:
      stream_name: ott-telemetry-live
      region: ap-south-1
    # bounded queue absorbs match-start bursts; retry protects against
    # transient backend slowdowns instead of applying backpressure upstream
    encoding:
      name: otlp
    max_records_per_batch: 500

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [awskinesis]

Kafka producer tuning for a telemetry firehoseproperties

# Producer tuned for high-throughput telemetry during a live match
acks=1
linger.ms=25
batch.size=1048576
compression.type=zstd
max.in.flight.requests.per.connection=5
buffer.memory=536870912
# partitions set the parallelism ceiling; size for peak RPS / per-partition throughput
# topic: ott-telemetry-live  partitions=256  replication.factor=3

Alert on telemetry stream consumer lag (leading indicator)yaml

groups:
  - name: telemetry-backpressure.rules
    rules:
      - alert: TelemetryConsumerLagHigh
        expr: |
          max by (consumergroup) (
            kafka_consumergroup_lag{topic="ott-telemetry-live"}
          ) > 500000
        for: 2m
        labels:
          severity: page
          team: observability
        annotations:
          summary: "Telemetry ingestion falling behind during live event"
          description: "Consumer lag > 500k; backend cannot keep up. Scale gateway/consumers before spans are dropped."

Production configs and runbooks are a Pro feature.

250K-500K RPS Live Match Architecture