Secrets

What is SOPS + OpenBao?

SOPS (Secrets OPerationS) is a tool for encrypting secret files using age, PGP, or cloud KMS keys — allowing encrypted secrets to be safely committed to a public git repository. OpenBao (a community fork of HashiCorp Vault) is a secrets management server that stores runtime credentials and serves them to pods via the Secrets Store CSI Driver.

Why This Approach?

SOPS with age encryption keeps bootstrap secrets version-controlled and reproducible without any external service dependency. OpenBao handles runtime secrets at pod startup — no credentials ever appear in Kubernetes manifests, git history, or CDK8s-generated YAML.

How It's Used Here

Bootstrap secrets (OpenBao unseal key, Cloudflare API token) are age-encrypted in secrets/bootstrap.sops.yaml and applied once with just create-secrets. All other app credentials live in OpenBao's KV store and are mounted into pods as files via the CSI driver at runtime, keeping CDK8s manifests completely secret-free.

Architecture

Secrets management uses a two-tier approach:

1. Bootstrap secrets — encrypted with SOPS/age, committed safely to git, applied once with just create-secrets
2. Runtime secrets — stored in OpenBao (Vault fork, MPL-2.0), mounted into pods as files via the Secrets Store CSI Driver

secrets/bootstrap.sops.yaml  (age-encrypted, safe in git)
  └── just create-secrets
        ├── openbao/openbao-unseal-key    (Prune=false)
        └── cert-manager/cloudflare-api-token  (Prune=false)

OpenBao (ns: openbao, port 8200)
  ├── KV v2 at secret/
  │     ├── secret/data/grafana   ADMIN_PASSWORD
  │     ├── secret/data/harbor    HARBOR_ADMIN_PASSWORD
  │     ├── secret/data/n8n       ENCRYPTION_KEY
  │     └── secret/data/netbird   NETBIRD_SETUP_KEY
  └── Kubernetes Auth method
        └── per-app roles → bound to app ServiceAccount + namespace

Secrets Store CSI Driver (ns: kube-system)
  └── SecretProviderClass (per app ns)

What happens at pod start

 flowchart TB
    SPC["SecretProviderClass<br/>(app namespace)"]
    SA["Pod ServiceAccount<br/>token"]
    BAO["OpenBao<br/>Kubernetes auth → role → policy"]
    VOL["CSI volume<br/>mounted in the pod"]
    FILE["/mnt/secrets/KEY<br/>Pattern A — file only"]
    SEC["k8s Secret<br/>Pattern B — secretObjects"]
    ENV["Container env var"]

    SA --> BAO
    SPC --> BAO
    BAO -->|"secret value"| VOL
    VOL --> FILE
    VOL -->|"only if secretObjects is set"| SEC
    SEC --> ENV

The arrow that catches people is the one from the volume: the mount is what drives everything. A SecretProviderClass on its own fetches nothing, and a secretObjects block on its own creates nothing. If no pod mounts the volume, the k8s Secret never appears — which is why Harbor runs a pause container whose only job is to hold the mount open.

Pattern A — File-only (no k8s Secret)

The secret is mounted as a file at /mnt/secrets/<KEY> and the app reads it via an env var naming that path — GF_SECURITY_ADMIN_PASSWORD__FILE, for instance.

No k8s Secret is created, so the value never appears in kubectl get secret. Prefer this whenever the app can read a file.

Pattern B — secretObjects sync (k8s Secret created)

Used by Harbor, n8n, NetBird, and Grafana's OIDC client secret.

The CSI volume mount triggers the SecretProviderClass secretObjects block, which creates a k8s Secret in the app's namespace. Needed when a Helm chart only accepts an existingSecret reference, or when the app can read the value only from an env var rather than a file.

The CSI volume mount is required to trigger the sync — if no pod mounts the volume, the k8s Secret is never created. This is the single most common cause of "the Secret was never created".

Harbor's chart has no extraVolumes support at all, so a dedicated secret-sync Deployment running a pause container mounts the CSI volume purely to trigger the sync.

An app can use both: Grafana's admin password is file-only (Pattern A) while its OIDC client secret is synced (Pattern B), because Grafana reads that one from a GF_ env var.

Bootstrap Secrets

Only two Secrets are created by the bootstrap script and never managed by Argo CD:

Both carry argocd.argoproj.io/sync-options: Prune=false so Argo CD never deletes them.

SOPS + age Setup

First-Time Setup

# 1. Install tools
brew install age sops

# 2. Generate age key pair (back up this file!)
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt
# Output: Public key: age1abc123...

# 3. Add to shell profile (REQUIRED — sops 3.12+ does not auto-discover)
echo 'export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt"' >> ~/.zshrc
source ~/.zshrc

# 4. Register public key in .sops.yaml at repo root
# creation_rules:
#   - path_regex: ^secrets/
#     age: age1abc123...

# 5. Populate the bootstrap secrets file
sops secrets/bootstrap.sops.yaml   # opens $EDITOR, re-encrypts on save

Day-to-Day Commands

# Edit encrypted file in-place
sops secrets/bootstrap.sops.yaml

# Create/update bootstrap k8s Secrets
just create-secrets

OpenBao Setup (one-time)

# 1. Deploy OpenBao + unseal it
just openbao-init        # initialises, saves root token to /tmp/openbao-init.json

# 2. Configure K8s auth, policies, roles, placeholder secrets
just openbao-setup       # runs scripts/openbao-setup.sh

# 3. Replace placeholder secrets with real values
ROOT_TOKEN=$(python3 -c "import json; print(json.load(open('/tmp/openbao-init.json'))['root_token'])")
kubectl exec -n openbao openbao-0 -- env BAO_TOKEN=$ROOT_TOKEN \
  bao kv put -mount=secret grafana  ADMIN_PASSWORD=<real>
kubectl exec -n openbao openbao-0 -- env BAO_TOKEN=$ROOT_TOKEN \
  bao kv put -mount=secret harbor   HARBOR_ADMIN_PASSWORD=<real>
kubectl exec -n openbao openbao-0 -- env BAO_TOKEN=$ROOT_TOKEN \
  bao kv put -mount=secret n8n      ENCRYPTION_KEY=<real>
kubectl exec -n openbao openbao-0 -- env BAO_TOKEN=$ROOT_TOKEN \
  bao kv put -mount=secret netbird  NETBIRD_SETUP_KEY=<real>

Apps and Their Secret Paths

n8n DB password is not in OpenBao — it is auto-managed by the CloudNativePG operator (n8n-pg-app Secret).

CDK8s Generates Zero Secrets

The CI pipeline synthesizes CDK8s manifests to the v0.1.7-manifests branch. It requires zero GitHub Actions secrets — CDK8s never generates any Secret resources. All runtime secrets are pulled by the in-cluster CSI driver at mount time.