NetBird Peer

What is the NetBird Peer?

The NetBird peer is an in-cluster WireGuard client that connects to the NetBird VPN mesh and acts as a routing peer for the cluster's LAN subnet (192.168.1.0/24). It runs with hostNetwork: true on any available worker (no nodeSelector — floats for resilience) and enables the Hetzner VPS (Bifrost) to reach cluster services via WireGuard, making public service routing through Traefik possible.

Bifrost Traefik
    │  proxy to 192.168.1.220:80
    ↓
netbird-agent (Bifrost · wt0: 100.109.47.211)
    │  WireGuard tunnel via relay
    ↓
netbird-peer-0 (any worker · wt0: 100.109.244.71)
    │  kernel IP forward: wt0 → ens18
    │  CILIUM_POST_nat MASQUERADE: src → <worker node IP>
    ↓
192.168.1.220 (Cilium Gateway · another node's ens18)
    │  Cilium BPF L7LB DNAT → Envoy :13507
    ↓
Grafana / other cluster pod

Source: workloads/networking/netbird_peer.go

Why a Kubernetes StatefulSet?

The in-cluster NetBird peer is deployed as a StatefulSet (not a Deployment) with a persistent PVC for /var/lib/netbird/:

• /var/lib/netbird/ stores the WireGuard private key and peer registration state
• Without persistence, every pod restart generates a new private key → new peer registration in NetBird Management → accumulating duplicate peers in the UI
• StatefulSet + PVC ensures the same peer identity is reused across restarts

A simple Deployment would create a new peer registration on every restart (e.g., every Argo CD sync that changes the pod spec), filling the NetBird Management UI with ghost peers.

Configuration

Critical: PVC mount path is /var/lib/netbird/ — not /etc/netbird/. NetBird stores its private key at /var/lib/netbird/ regardless of what NB_CONFIG points to. Mounting at the wrong path means the key is never persisted.

Secrets (OpenBao)

Pattern B (secretObjects sync). NETBIRD_SETUP_KEY is fetched from OpenBao (secret/data/netbird) and synced into the netbird-setup-key k8s Secret.

The setup key is only used on first registration. Once the peer is registered, subsequent restarts reuse the private key from the PVC and ignore the setup key.

MASQUERADE initContainer

The StatefulSet includes an initContainer that adds an iptables rule before the NetBird agent starts:

iptables -t nat -C POSTROUTING -o ens18 -d 192.168.1.0/24 -j MASQUERADE 2>/dev/null \
  || iptables -t nat -A POSTROUTING -o ens18 -d 192.168.1.0/24 -j MASQUERADE

The rule matches on the outbound interface (-o ens18), not a source range: anything leaving the node toward the cluster LAN is source-NAT'd to that node's own IP, so replies come back via normal LAN routing instead of trying to find the WireGuard overlay. The -C check makes it idempotent across pod restarts.

In practice the actual NAT is performed by Cilium's CILIUM_POST_nat BPF chain, not the raw iptables rule. Both coexist without conflict.

Cilium Constraint — wt0 Must NOT Be in Devices

Do not add wt0 to Cilium's devices list in core/platform/cilium.go. WireGuard interfaces are NOARP/POINTOPOINT with no Ethernet header — Cilium's cil_from_netdev TC BPF silently drops all packets without monitor events, breaking all traffic from Bifrost.

wt0 must never appear in Cilium devices. Traffic from wt0 reaches Cilium BPF via the node's ens18 after kernel IP forwarding and MASQUERADE. See Network Flow — Why wt0 is NOT in Cilium Devices.

Route Configuration

The peer registers in NetBird but does not automatically advertise routes. Routes must be configured in the NetBird Management UI:

NetBird Management UI → Network → Routes → Add Route:

Without this route configuration, the Bifrost VPS cannot reach the cluster's LAN services even though the WireGuard tunnel is up.

Troubleshooting

Peer not connecting

# Check pod is running
kubectl get pods -n netbird -o wide

# Check NetBird agent status
kubectl exec -n netbird netbird-peer-0 -- netbird status
# Expect: Management: Connected, Peers count: 1/1 Connected

# Check NetBird agent logs for errors
kubectl logs -n netbird netbird-peer-0 | tail -30

# Verify management URL is reachable from the pod
kubectl exec -n netbird netbird-peer-0 -- \
  wget -qO- --timeout=5 https://netbird.madhan.app/api/v1/peers 2>&1 | head -5

Route shows "Networks: -" on bifrost-agent

The peer is connected but not advertising the route. Either:

1. The Network Route hasn't been created in NetBird UI → Network Routes
2. The route references a stale/disconnected peer — delete old entries and re-assign to the current peer

# Verify from the peer side
kubectl exec -n netbird netbird-peer-0 -- netbird status
# Look for: Networks: 192.168.1.0/24

# Verify from Bifrost
ssh root@178.156.199.250 'docker exec netbird-agent netbird routes list'
# Expect: 192.168.1.0/24  Status: Selected

504 from public services but tunnel is up

The route is connected but traffic doesn't reach the cluster backend. Check the Cilium device configuration first:

# Confirm wt0 is NOT in Cilium's device list on the routing peer's node
NODE=$(kubectl get pod -n netbird netbird-peer-0 -o jsonpath='{.spec.nodeName}')
CILIUM_POD=$(kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium-agent \
  -o wide | grep "$NODE" | awk '{print $1}')
kubectl exec -n kube-system $CILIUM_POD -c cilium-agent -- cilium-dbg status | grep Devices
# Expect ens18 and eth0 only — wt0 must never appear
# If wt0 appears: update core/platform/cilium.go and run just core platform up

# Bifrost side: agent connected, route selected, kernel route present
ssh root@178.156.199.250 'docker exec netbird-agent netbird status'
# Expect: Management: Connected, Peers count: 1/1 Connected
ssh root@178.156.199.250 'docker exec netbird-agent netbird routes list'
# Expect: 192.168.1.0/24  Status: Selected
ssh root@178.156.199.250 'ip route show table 7120'
# Expect: 192.168.1.0/24 dev wt0

# Test connectivity to the Cilium LB from Bifrost
ssh root@178.156.199.250 \
  'curl -sv -H "Host: grafana.madhan.app" --connect-timeout 5 http://192.168.1.220/'
# Expect: HTTP/1.1 302 Found

# If that times out but a NodePort on another worker answers, the problem is on
# the routing peer's node — check Cilium there.
ssh root@178.156.199.250 \
  'curl -sv -H "Host: grafana.madhan.app" --connect-timeout 3 http://192.168.1.222:32601/'

wt0 shows 0 bytes — tunnel dead

NB_SKIP_SOCKET_MARK must not be set on the StatefulSet. It disables the socket fwmark, so management traffic loops back through the WireGuard tunnel and the relay connection never establishes.

netbird-agent not running on Bifrost

ssh root@178.156.199.250 'docker ps | grep netbird-agent'
ssh root@178.156.199.250 'NB_BIFROST_SETUP_KEY=$(grep NB_BIFROST_SETUP_KEY /etc/bifrost/.secrets.env | cut -d= -f2) docker compose -f /etc/bifrost/docker-compose.yml up -d netbird-agent'

MASQUERADE rule not taking effect

The iptables MASQUERADE rule requires the pod to have NET_ADMIN capability. Check the initContainer ran:

kubectl describe pod -n netbird netbird-peer-0 | grep -A5 "Init Containers"
# setup-iptables should show: State: Terminated, Reason: Completed

# Check the rule is installed
kubectl exec -n netbird netbird-peer-0 -- \
  iptables -t nat -L POSTROUTING -n -v | grep -E "MASQUERADE|CILIUM"
# CILIUM_POST_nat chain should show high packet count
# MASQUERADE rule (backup) shows 0 packets — this is normal with Cilium BPF active