PromCon EU 2026Talk abstract
A Prometheus-Native Approach to Global Querying
Gardener's PromQL Querier is a working prototype for a distributed query planner for Prometheus, targeting environments that need a global view across many independent Prometheus instances. It answers queries as if it were a single Prometheus instance, fanning out partial queries to downstream instances and combining the results.
Its philosophy is to rely solely on native Prometheus capabilities by querying downstreams through the HTTP API and invoking the Prometheus PromQL engine. This makes it a distributed query planner rather than a distributed engine.
Each downstream is configured with a unique set of virtual labels that make time series distinguishable across downstreams, for example env="prod",region="europe" or env="dev",region="america". Since virtual labels do not exist in the downstream TSDBs, the Querier injects them at execution time.
When queried, it parses the PromQL AST and plans its distributed execution. It identifies the expressions that can be pushed down and determines which downstreams they target based on the virtual label matchers in the query. The query is executed by fanning out those expressions and evaluating the remainder through the Prometheus PromQL engine over the partial results.
The talk covers how Gardener's PromQL Querier analyzes PromQL constructs for push-down eligibility and how queries are rewritten to combine the partial results. Finally, it discusses the limitations of this approach.
Speakers
