Skip to content

Grafana

Grafana is the UI over VictoriaMetrics and VictoriaLogs. It is the one public workload, with its own OIDC login. Upstream docs.

Namespacegrafana
Sourcegrafana.go
ExposurePublic · grafana.madhan.app (in publicServices, SkipAuth)
Storage10Gi RWX on the default longhorn class
AuthAuthentik OIDC (GitHub login); login form disabled

SkipAuth leaves out Bifrost’s Authentik ForwardAuth, since Grafana’s own OIDC login would make it a second prompt.

Grafana main dashboard with node resource panels

Non-admin SSO users never see logs.

OrgIDWhoRoleDatasourcesDashboards
Main Org.1Authentik group grafana-adminsAdminVictoriaMetrics, VictoriaLogsEvery sidecar ConfigMap
Public2Every other SSO userViewerVictoriaMetricsKubernetes Views Global (15757), NVIDIA DCGM (12239)

org_mapping (grafana-admins:1:Admin *:2:Viewer) is re-evaluated on every login. Public-org dashboards are downloaded from grafana.com by the chart’s init container, so the pod needs outbound internet to start.

SettingValueWhy
server.root_urlhttps://grafana.madhan.appTLS ends at Bifrost, so without it Grafana builds an http:// OAuth callback that Authentik rejects
auth.generic_oauth.client_idgrafana-homelabMatches the provider in authentik.go
auth.disable_login_formtrueSSO only
Admin password$__file{/mnt/secrets/ADMIN_PASSWORD}Read from the CSI mount; the distroless image does not support GF_*__FILE
OAuth secretenvFromSecret: grafana-oauth-secretCSI-synced from OpenBao key OAUTH_CLIENT_SECRET
VictoriaLogs URL…:9428/selectThe Loki plugin appends /loki/api/v1/… itself
logsVolumeEnabledfalseVictoriaLogs has no Loki volume API
Dashboard sidecarlabel grafana_dashboard: "1", all namespacesAny chart can ship a dashboard as a ConfigMap; folder from grafana_folder
defaultCurlOptions-sfThe chart default adds -k, which skips TLS verification

OpenBao keys at secret/grafana: ADMIN_PASSWORD, OAUTH_CLIENT_SECRET, seeded by just openbao-setup.

On a fresh cluster, create the Public org before Argo CD syncs the two-org config: Grafana refuses to start while provisioning references a missing org.

Terminal window
just grafana-init # creates org "Public", asserts it got ID 2; idempotent

Grant admin by adding the user to grafana-admins in Authentik, Directory → Groups.

Break-glass when SSO is down: reset the local admin password and use the API with basic auth.

Terminal window
kubectl exec -n grafana deploy/grafana -c grafana -- grafana-cli admin reset-admin-password '<new-password>'
curl -u admin https://grafana.madhan.app/api/org

Login fails with redirect_uri mismatch from outside the LAN

Section titled “Login fails with redirect_uri mismatch from outside the LAN”

root_url is unset or wrong, so the callback is http://. Check grafana.ini in the rendered values.

grafana-oauth-secret is missing because the CSI mount failed. Check OpenBao is unsealed and the grafana role exists:

Terminal window
kubectl get secret grafana-oauth-secret -n grafana
kubectl describe pod -n grafana -l app.kubernetes.io/name=grafana | grep -A5 openbao-secrets

The dashboard download init container cannot reach grafana.com. Check the node’s outbound DNS and internet.