From b8b95eb91852bddfdcfe01102d722235511f617d Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 19 Aug 2025 08:44:16 -0400 Subject: [PATCH] docs: warn against enabling Prometheus metrics if not in use (#26560) The go-metrics library retains Prometheus metrics in memory until expiration, but the expiration logic requires that the metrics are being regularly scraped. If you don't have a Prometheus server scraping, this leads to ever-increasing memory usage. In particular, high volume dispatch workloads emit a large set of label values and if these are not eventually aged out the bulk of Nomad server memory can end up consumed by metrics. --- website/content/docs/configuration/telemetry.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/website/content/docs/configuration/telemetry.mdx b/website/content/docs/configuration/telemetry.mdx index 9f165bc66..3ad586cbc 100644 --- a/website/content/docs/configuration/telemetry.mdx +++ b/website/content/docs/configuration/telemetry.mdx @@ -149,8 +149,10 @@ telemetry { These `telemetry` parameters apply to [Prometheus]. -- `prometheus_metrics` `(bool: false)` - Specifies whether the agent should - make Prometheus formatted metrics available at `/v1/metrics?format=prometheus`. +- `prometheus_metrics` `(bool: false)` - Specifies whether the agent should make + Prometheus formatted metrics available at `/v1/metrics?format=prometheus`. + Nomad's Prometheus client retains metrics in memory unless scraped, so you + should not enable this field unless you are collecting metrics via Promtheus. ### `circonus` (Apica)