Skip to content

Troubleshooting

Find the symptom, then follow the link. Component pages carry the detailed fixes.

SymptomLikely causeWhere to look
504 Gateway Timeout on a public URLTraefik cannot reach the cluster: NetBird tunnel or routeNetBird peer
Every LoadBalancer IP goes dark after a Cilium restartStale L2 announcement lease; the holder renews but stops answering ARPjust gateway-repair-l2 — Repairs
A *.madhan.app host resolves but nothing answersService is LAN-only and you are off the LAN and off the VPNExposure
TLS certificate errors on a serviceWildcard cert, or Cilium’s copy of it in cilium-secretscert-manager
Argo CD UI loads but lists no applicationsStale aggregated APIService breaks API discovery; kubectl api-resources shows errorsArgo CD
SymptomLikely causeWhere to look
GPU pod stays Pending with no clear reasonMissing the dedicated=ai:NoSchedule toleration; worker4 is taintedGPU
GPU pod runs but CUDA is invisibleMissing runtimeClassName: nvidiaGPU
Multi-Attach error for volumeRWO volume still held by the old pod during a rolloutHarbor
PVC stuck Pending, pod stuck ContainerCreatingLonghorn volume cannot attachLonghorn
No Longhorn volume on a node attaches after a Talos upgradeNew open-iscsi rejects a record its predecessor wrotejust longhorn-repair-iscsi — Repairs
Longhorn or CSI still down after DiskPressure clearedDaemonSet pods evicted under pressure stay Failed and are never replacedDelete the Failed pods — Repairs
Pod cannot find its Kubernetes SecretNothing mounts the CSI volume, so the sync never firesSecrets
ComfyUI returns 503It ships scaled to zerojust comfyui on — ComfyUI
SymptomLikely causeWhere to look
kubectl top says Metrics API not availableMetrics ServerMetrics Server
Metrics missing from GrafanaVMAgent is not scraping the targetVictoriaMetrics
Ollama cannot load a modelComfyUI holds VRAM; the error surfaces on OllamaGPU, just comfyui off
Grafana OIDC login loops or errorsroot_url / redirect URI mismatchGrafana
Falco UI never becomes readyRedis wedged with MISCONF: TTL and volume sizeFalco
App stuck OutOfSync in Argo CDSync wave, hook failure, or SSA ownership conflictArgo CD
A manual kubectl change reverted itselfselfHeal: true — intended; change it in gitArgo CD
SymptomLikely causeWhere to look
Node will not reboot after a Cilium problemUnmount blocks on an NFS share-manager ClusterIP it can no longer reachPower-cycle — Repairs
Talos upgrade hangs draining a nodeLonghorn and single-instance CNPG PDBs allow zero disruptionsjust talos-upgrade <ip> <schematic> false — Talos upgrade
Cluster health check fails between node upgradesStop and fix before continuingjust talos-health
Terminal window
just talos-health # cluster-level health
kubectl get applications -n argocd # what Argo CD thinks is wrong
kubectl api-resources 2>&1 >/dev/null # any output means broken discovery
kubectl get events -A --sort-by=.lastTimestamp | tail -30