Skip to content

VictoriaMetrics

VictoriaMetrics stores every metric in the cluster and answers PromQL for Grafana. One chart brings VMSingle, VMAgent, VMAlert, Alertmanager, node-exporter and kube-state-metrics. Upstream docs.

Namespacevictoria-metrics (privileged, for node-exporter)
Sourcevictoria_metrics.go
ExposureLAN · vmselect.madhan.app (VMUI), alertmanager.madhan.app
Storage100Gi on the default longhorn class, 30-day retention
AuthNone

Why VictoriaMetrics over Prometheus: VictoriaMetrics and VictoriaLogs.

SettingValueWhy
ModeVMSingle, vmcluster.enabled: falseOne cluster’s metrics do not justify the insert/select/storage split
fullnameOverridevm-stackThe default name pushes Alertmanager pod labels past 63 bytes. Endpoint: vmsingle-vm-stack.victoria-metrics.svc:8428
CRDsVM operator CRDs, plus ServiceMonitor and PrometheusRulecdk8s.NewHelm never renders a chart’s crds/; the operator converts ServiceMonitors
vmagent.selectAllByDefaulttrueScrapes every ServiceMonitor and PodMonitor cluster-wide, every 30s
OpenBao scrapeinlineScrapeConfig on /v1/sys/metricsOpenBao has no ServiceMonitor
Operator CRD cleanup hookdisabledArgo CD would run the pre-delete hook on every sync
Alertmanager receiverblackholeNo notification channel yet
kubeProxydisabledCilium replaces kube-proxy
Default dashboards and rulesenabledCreated by the chart’s sync job (an Argo CD PostSync hook), not rendered
GrafanadisabledDeployed separately as Grafana

The vmselect route redirects / to /vmui/. The hostname predates the move to single-node.

Add a real receiver under alertmanager.config.receivers in victoria_metrics.go to send notifications.

Terminal window
curl -s https://alertmanager.madhan.app/api/v2/alerts | jq .
kubectl port-forward -n victoria-metrics svc/vmagent-vm-stack 8429:8429 # then open /targets

The ServiceMonitor is absent or its selector misses the Service’s labels. Check /targets on VMAgent (above).

The volume is full. Raise the vmsingle storage request in victoria_metrics.go, or shorten retentionPeriod.