beginner ยท Module 1.2
Prometheus Architecture on Kubernetes
Learning Objectives
- Describe the Prometheus pull model and how service discovery finds OTT workloads on Kubernetes.
- Explain the roles of the scrape config, TSDB, and the WAL.
- Understand how ServiceMonitors (Prometheus Operator) wire targets to scrape jobs.
Key Concepts
Pull model + service discovery
Prometheus scrapes /metrics endpoints on a schedule. On Kubernetes it uses kubernetes_sd_configs (or ServiceMonitors via the Operator) to discover pods and services dynamically as OTT deployments scale up during a match.
TSDB and the WAL
Samples land in a write-ahead log then compact into immutable blocks. Retention and block size govern how much live-match history you can query locally before shipping to long-term storage.
ServiceMonitor wiring
A ServiceMonitor selects services by label and defines the port/path/interval, letting platform teams onboard new OTT microservices without editing the central scrape config by hand.
Info
Prometheus is a pull-based system; short-lived jobs push to a Pushgateway instead of being scraped.
Warning
A too-short scrape interval multiplied by thousands of match-time pods can overwhelm both the targets and the TSDB.
Knowledge Check
1. How does Prometheus collect metrics from OTT workloads by default?
2. What does a ServiceMonitor (Prometheus Operator) primarily do?
3. What is the purpose of the Prometheus write-ahead log (WAL)?