OpenBao

What is OpenBao?

OpenBao is an open-source, community-maintained fork of HashiCorp Vault (MPL-2.0). It provides secrets management with a fully Vault-compatible API — all vault CLI knowledge and tooling applies directly using the bao CLI.

Why OpenBao?

OpenBao's Kubernetes auth method allows pods to authenticate using their ServiceAccount token without any static credentials — the cluster itself is the identity provider. The Secrets Store CSI Driver integration mounts secrets directly into pods as files at startup, so no secret values appear in manifests, environment variables, or kubectl get secret output.

First-Time Setup

Initial bootstrap (init, unseal, K8s auth configuration, writing app secrets) is covered in Deployment Guide — Phase 2.

How It's Used Here

OpenBao runs as a standalone (single-node) server in the openbao namespace. Pods authenticate using the Kubernetes auth method — they present their ServiceAccount JWT token, OpenBao verifies it with the Kubernetes tokenreviews API, and returns a short-lived token.

Kubernetes auth roles:

Secret paths (KV v2):

Source: workloads/secrets/openbao.go

Secrets Patterns

Apps consume OpenBao secrets as mounted files, optionally synced to a k8s Secret. Both patterns are explained once in Secrets — the short version is that the CSI volume mount is what triggers the sync, so a SecretProviderClass with no pod mounting it produces nothing.

Configuration

Auto-Unseal Sidecar

OpenBao starts in a sealed state after every pod restart. An unseal sidecar container (part of the Helm values extraContainers) polls every 15 seconds and unseals with the key from the openbao-unseal-key Secret:

// workloads/secrets/openbao.go
"extraContainers": []map[string]any{
    {
        "name":  "unseal",
        "image": "openbao/openbao:<version>",   // matches the server image
        "command": []string{"sh", "-c", `
while true; do
  STATUS=$(bao status -format=json 2>/dev/null || echo '{"sealed":true}')
  if echo "$STATUS" | grep -q '"sealed":true'; then
    bao operator unseal "$OPENBAO_UNSEAL_KEY" 2>/dev/null || true
  fi
  sleep 15
done`},
    },
}

Why extraContainers and not extraInitContainers? Init containers must complete before the main container starts, but OpenBao must be running before it can accept an unseal request. A sidecar container runs alongside the main container and can poll until the server is ready.

The openbao-unseal-key Secret is created by just create-secrets from secrets/bootstrap.sops.yaml. It carries argocd.argoproj.io/sync-options: Prune=false so Argo CD never deletes it.

How It Connects

Pod starts
  → CSI volume mount → OpenBao CSI provider DaemonSet
  → OpenBao CSI provider: Kubernetes auth with pod SA token
  → OpenBao: verifies SA token via Kubernetes tokenreviews API
  → OpenBao: returns secret value(s)
  → CSI provider: writes secrets as files to pod filesystem
  → (Pattern B) CSI driver: creates/updates k8s Secret via secretObjects

Screenshots

Commands

The openbao-unseal-key bootstrap Secret stores only unseal-key — the root token is not persisted. Use the just recipes below to generate a temporary root token on demand (they handle the full generate-root ceremony automatically and revoke the token after use).

Read a secret (recommended)

just openbao-get secret/grafana
just openbao-get secret/grafana ADMIN_PASSWORD

Generate a root token for interactive use

ROOT_TOKEN=$(just openbao-token)

Revoke it when done:

just openbao-revoke $ROOT_TOKEN

Raw bao commands (after exporting ROOT_TOKEN)

# List all secrets
kubectl exec -n openbao openbao-0 -c openbao -- \
  env VAULT_TOKEN=$ROOT_TOKEN bao kv list secret/

# Patch a single key (non-destructive)
kubectl exec -n openbao openbao-0 -c openbao -- \
  env VAULT_TOKEN=$ROOT_TOKEN bao kv patch secret/grafana ADMIN_PASSWORD=newvalue

# Replace all keys (full put)
kubectl exec -n openbao openbao-0 -c openbao -- \
  env VAULT_TOKEN=$ROOT_TOKEN bao kv put secret/grafana ADMIN_PASSWORD=value OAUTH_CLIENT_SECRET=value

# Check auth roles
kubectl exec -n openbao openbao-0 -c openbao -- \
  env VAULT_TOKEN=$ROOT_TOKEN bao read auth/kubernetes/role/grafana

Check OpenBao status

kubectl exec -n openbao openbao-0 -- bao status

Troubleshooting

Sealed After Restart

Symptoms: Pods that depend on OpenBao secrets fail to start; OpenBao pod shows sealed=true.

Diagnosis:

kubectl exec -n openbao openbao-0 -- bao status

Fix: The unseal sidecar should handle this automatically. If the sidecar is not working:

# Check sidecar logs
kubectl logs -n openbao openbao-0 -c unseal

# Manual unseal (get key from bootstrap secret)
UNSEAL_KEY=$(kubectl get secret openbao-unseal-key -n openbao -o jsonpath='{.data.unseal-key}' | base64 -d)
kubectl exec -n openbao openbao-0 -- bao operator unseal "$UNSEAL_KEY"

403 Permission Denied

Symptoms: CSI mount fails with permission denied or 403 Forbidden.

Diagnosis:

# Check the OpenBao CSI provider logs on the affected node
kubectl logs -n openbao -l app.kubernetes.io/name=openbao-csi-provider

# Test the auth role manually
kubectl exec -n openbao openbao-0 -- bao read auth/kubernetes/role/<app-name>

Fix: The Kubernetes auth role may not include the pod's ServiceAccount. Run just openbao-setup to re-apply roles, or manually add the SA:

kubectl exec -n openbao openbao-0 -- env BAO_TOKEN=$ROOT_TOKEN \
  bao write auth/kubernetes/role/<app> \
    bound_service_account_names=<sa-name> \
    bound_service_account_namespaces=<namespace> \
    policies=<policy-name> \
    ttl=1h

CSI Mount Failing

Symptoms: Pod stuck in ContainerCreating with MountVolume.SetUp failed.

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>
# Look for: failed to get secretproviderclass

kubectl get secretproviderclass -n <namespace>
kubectl describe secretproviderclass <name> -n <namespace>

Fix: Ensure the SecretProviderClass exists in the same namespace as the pod. CDK8s should create it — check if Argo CD has synced the namespace.