OpenTelemetry
Two OpenTelemetry Collector releases ship metrics to VictoriaMetrics and logs to VictoriaLogs. Upstream docs.
| Namespace | opentelemetry (privileged: hostPath and hostPorts) |
| Source | otel_collector.go |
| Exposure | — |
| Storage | hostPath /var/lib/otelcol on each node (log read offsets) |
| Auth | — |
How it fits
Section titled “How it fits”| Release | Mode | Receivers | Exports |
|---|---|---|---|
otel-agent | DaemonSet, every node (tolerations: Exists) | filelog (/var/log/pods), kubeletstats, hostmetrics | Logs → VictoriaLogs, metrics → VictoriaMetrics |
otel-gateway | Deployment, 1 replica | k8s_cluster, k8sobjects (events) | Metrics → VictoriaMetrics, events as logs → VictoriaLogs |
Falco writes JSON alerts to stdout, so the agent’s filelog receiver picks them up with every other container log.
Configuration
Section titled “Configuration”| Setting | Value | Why |
|---|---|---|
| Image | otel/opentelemetry-collector-contrib | Carries the Kubernetes receivers |
kubeletstats.endpoint | https://${env:K8S_NODE_IP}:10250, insecure_skip_verify | The preset uses the node name, which does not resolve in-cluster on Talos |
file_storage extension | /var/lib/otelcol on a hostPath | Restarts resume from saved offsets instead of re-reading old logs |
prometheusremotewrite | vmsingle-vm-stack…:8428/api/v1/write, resource_to_telemetry_conversion | Resource attributes such as k8s.pod.name become labels |
k8sattributes.passthrough (gateway) | true | Events have no pod IP or UID; the default drops all of them |
memory_limiter | 80%, spike 25% | |
batch | 10s |
Troubleshooting
Section titled “Troubleshooting”Logs or metrics missing
Section titled “Logs or metrics missing”Check every node has an agent and the exporters log no errors:
kubectl get pods -n opentelemetry -o widekubectl logs -n opentelemetry -l app.kubernetes.io/instance=otel-agent --tail=50 | grep -i errorKubernetes events missing from VictoriaLogs
Section titled “Kubernetes events missing from VictoriaLogs”k8sattributes.passthrough is not true on the gateway, so every event is dropped before export.