Skip to content

NVIDIA device plugin

The standalone NVIDIA device plugin advertises nvidia.com/gpu on worker4 and splits the card five ways by time-slicing. Upstream docs.

Namespacenvidia-gpu-operator (privileged)
Sourcenvidia_gpu_operator.go
Exposure—
Storage—
Auth—

The full GPU Operator is not used: on Talos the driver and container toolkit come from system extensions, and the operator’s validator fails on Talos library paths.

flowchart LR
  ext["Talos extensions<br/>kernel modules + container toolkit"] --> rc["RuntimeClass nvidia"]
  nfd["NFD + GFD"] -->|"nvidia.com/gpu.present=true"| dp["device plugin<br/>nvidia.com/gpu: 5"]
  dp --> pod["GPU pod<br/>runtimeClassName: nvidia"]
  rc --> pod
  dcgm["DCGM exporter"] -->|ServiceMonitor| vm["VictoriaMetrics"]
SettingValueWhy
RuntimeClassnvidia (handler nvidia)Routes pods through nvidia-container-runtime, the only way to inject NVML on Talos (libraries sit in a squashfs)
deviceDiscoveryStrategynvmlauto probes standard paths that do not exist on Talos
plugin.deviceListStrategyenvvarCDI hostPath mounts fail on Talos’s non-standard library paths
timeSlicing.replicas5Ollama, ComfyUI, a notebook, a training job and a Katib trial
NFD, GFDenabledGFD sets nvidia.com/gpu.present=true, the selector every GPU pod uses
Tolerationsdedicated=ai:NoScheduleworker4 carries that taint
DCGM exporterGPU node only, serviceMonitor.enabledUtilisation, VRAM, temperature and power, shown on the NVIDIA DCGM dashboard in Grafana

What a GPU pod needs, and why the five units share one VRAM pool: Scheduling a GPU pod.

Check the plugin runs on worker4 and the allocatable count:

Terminal window
kubectl get pods -n nvidia-gpu-operator -o wide
kubectl describe node k8s-worker4 | grep -A6 Allocatable # expect nvidia.com/gpu: 5

The kernel modules are not loaded, usually because the node booted the base image instead of the GPU schematic.

Terminal window
talosctl --talosconfig ~/.talos/config dmesg --nodes 192.168.1.224 | grep nvidia

Upgrade the node to the GPU schematic with just talos-upgrade 192.168.1.224 gpu.