Skip to content

OpenTelemetry

Two OpenTelemetry Collector releases ship metrics to VictoriaMetrics and logs to VictoriaLogs. Upstream docs.

Namespaceopentelemetry (privileged: hostPath and hostPorts)
Sourceotel_collector.go
Exposure—
StoragehostPath /var/lib/otelcol on each node (log read offsets)
Auth—
ReleaseModeReceiversExports
otel-agentDaemonSet, every node (tolerations: Exists)filelog (/var/log/pods), kubeletstats, hostmetricsLogs → VictoriaLogs, metrics → VictoriaMetrics
otel-gatewayDeployment, 1 replicak8s_cluster, k8sobjects (events)Metrics → VictoriaMetrics, events as logs → VictoriaLogs

Falco writes JSON alerts to stdout, so the agent’s filelog receiver picks them up with every other container log.

SettingValueWhy
Imageotel/opentelemetry-collector-contribCarries the Kubernetes receivers
kubeletstats.endpointhttps://${env:K8S_NODE_IP}:10250, insecure_skip_verifyThe preset uses the node name, which does not resolve in-cluster on Talos
file_storage extension/var/lib/otelcol on a hostPathRestarts resume from saved offsets instead of re-reading old logs
prometheusremotewritevmsingle-vm-stack…:8428/api/v1/write, resource_to_telemetry_conversionResource attributes such as k8s.pod.name become labels
k8sattributes.passthrough (gateway)trueEvents have no pod IP or UID; the default drops all of them
memory_limiter80%, spike 25%
batch10s

Check every node has an agent and the exporters log no errors:

Terminal window
kubectl get pods -n opentelemetry -o wide
kubectl logs -n opentelemetry -l app.kubernetes.io/instance=otel-agent --tail=50 | grep -i error

Kubernetes events missing from VictoriaLogs

Section titled “Kubernetes events missing from VictoriaLogs”

k8sattributes.passthrough is not true on the gateway, so every event is dropped before export.