Troubleshooting
Start from the symptom. Every entry links to the page with the full runbook.
Twenty-odd pages carry a Troubleshooting section, which is only useful if you already know which component is at fault. Start here instead — find the symptom, follow the link.
Something is unreachable
| Symptom | Likely cause | Where to look |
|---|---|---|
504 Gateway Timeout on a public URL | Traefik cannot reach the cluster — NetBird tunnel or route | NetBird Peer |
| Every LoadBalancer IP goes dark after a Cilium restart | Stale L2 announcement lease; the holder renews but stops answering ARP | Upgrade Guide, then just gateway-repair-l2 |
A *.madhan.app host resolves but nothing answers | Service is LAN-only and you are off the LAN and off the VPN | Service Access |
| TLS certificate errors on a service | Wildcard cert or the Cilium secret mirror | cert-manager |
| Argo CD UI loads but lists no applications | Usually a stale aggregated APIService breaking API discovery — check kubectl api-resources for errors | Argo CD |
A pod will not start
| Symptom | Likely cause | Where to look |
|---|---|---|
GPU pod stays Pending with no clear reason | Missing the dedicated=ai toleration — worker4 is tainted | GPU |
| GPU pod runs but CUDA is invisible | Missing runtimeClassName: nvidia | GPU |
Multi-Attach error for volume | RWO volume still held by the old pod during a rollout | Harbor |
PVC stuck Pending, pod stuck ContainerCreating | Longhorn volume cannot attach | Longhorn |
| No Longhorn volume on a node will attach after a Talos upgrade | New open-iscsi rejects a record its predecessor wrote | just longhorn-repair-iscsi — Upgrade Guide |
| Pod cannot find its k8s Secret | Nothing mounts the CSI volume, so the sync never fires | Secrets |
| ComfyUI returns 503 | It ships scaled to zero | just comfyui on — ComfyUI |
Something is missing or wrong
| Symptom | Likely cause | Where to look |
|---|---|---|
kubectl top says Metrics API not available | Metrics Server | Metrics Server |
| Metrics missing from Grafana | VMAgent is not scraping the target | VictoriaMetrics |
| Ollama cannot load a model | ComfyUI is holding VRAM, and the error surfaces on Ollama | GPU |
| Grafana OIDC login loops or errors | root_url / redirect URI mismatch | Grafana |
| Falco UI never becomes ready | Redis wedged with MISCONF — TTL and volume size | Falco |
App stuck OutOfSync in Argo CD | Sync wave, hook failure, or SSA ownership conflict | Argo CD |
A manual kubectl change reverted itself | selfHeal=true — this is intended | Argo CD |
A node is in trouble
| Symptom | Likely cause | Where to look |
|---|---|---|
| Node will not reboot after a Cilium problem | Unmount blocks on an NFS share-manager ClusterIP it can no longer reach | Upgrade Guide — powercycle |
| Talos upgrade hangs draining a node | Longhorn and single-instance CNPG PDBs allow zero disruptions | just talos-upgrade <ip> <schematic> false |
| Cluster health check fails between node upgrades | Stop and fix before continuing | just talos-health |
First commands
just talos-health # cluster-level health
kubectl get applications -n argocd # what Argo CD thinks is wrong
kubectl api-resources 2>&1 >/dev/null # any output here means broken discovery
kubectl get events -A --sort-by=.lastTimestamp | tail -30