Configure rules for aggregating scraped data into new time series
Though not a problem in our example, queries that aggregate over thousands of
time series can get slow when computed ad-hoc. To make this more efficient,
Prometheus can prerecord expressions into new persisted
time series via configuredrecording rules. Let's say we are interested in
recording the per-second rate of cpu time (node_cpu_seconds_total) averaged
over all cpus per instance (but preserving thejob,instanceandmodedimensions) as measured over a window of 5 minutes. We could write this as:
avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
Try graphing this expression.
To record the time series resulting from this expression into a new metric
calledjob_instance_mode:node_cpu_seconds:avg_rate5m, create a file
with the following recording rule and save it asprometheus.rules.yml:
groups:-name:cpu-noderules:-record:job_instance_mode:node_cpu_seconds:avg_rate5mexpr:avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))
To make Prometheus pick up this new rule, add arule_filesstatement in yourprometheus.yml. The config should now
look like this: