Prometheus logo PromCon 2018

The Prometheus conference — August 09 - 10 in Munich

Talk abstract

Bomb Squad: Containing the Cardinality Explosion

Speaker: Cody Boggs

Series cardinality can be a large foot-gun that can take out a Prometheus instance quickly. It can have similar impacts to downstream systems when using remote write. Finding the offending series and recovering from such an event can cause a large downtime for your Prometheus installation. In this talk I will show an implementation of a sidecar to Prometheus that automatically detects and squelches series whose cardinality is growing quickly. Using a combination of recording rules, dynamic metric relabel configs, and alerting, we can detect, throttle, and alert on any explosion in progress.


Back to schedule