Setting Retention for Kube Prometheus Stack

When deploying the prometheus-community/kube-prometheus-stack in a k8s cluster, the default retention for Prometheus scrapped data is 10d at the time of this writing or Helm chart kube-prometheus-stack-69.4.1. As a result, the longhorn storage is filled up fast, and caused the Prometheus to crash.

So I’d decided to change the retention period to 3 days, and also set retention size to 4GB. To do this, update the values file for the Helm chart with the following values:

prometheus:
  prometheusSpec:
    retention: 3d
    retentionSize: 4GB

Apply the settings:

$ helm upgrade prometheus prometheus-community/kube-prometheus-stack \
     --namespace monitoring \
     -f kube-prometheus-stack-values.yaml

To confirm the change, one way to do this is to check the status of “Runtime & build information” on the Prometheus dashboard.

Start time	2025-02-24T23:26:51Z
Working directory	/prometheus
Configuration reload	Successful
Last successful configuration reload	2025-02-24T23:27:02Z
WAL corruptions	0
Goroutines	300
GOMAXPROCS	1
GOMEMLIMIT	3445512192
GOGC	75
GODEBUG	
Storage retention	3d or 4GiB

Reference:

• [kube-prometheus-stack] Prometheus Retention value not respected #2703