Secrets Flow

Two-Tier Model

All secrets are managed by exactly two systems. The split is intentional — bootstrap secrets are needed to bootstrap the cluster itself, and the cluster hosts the runtime secrets platform:

CDK8s generates zero Secret resources. The CI pipeline needs zero GitHub Actions secrets.

Architecture Diagram

 flowchart TB
    subgraph SOPSFILE["secrets/bootstrap.sops.yaml<br/>age-encrypted · safe to commit to git"]
        S1["CLOUDFLARE_API_TOKEN<br/>HCLOUD_TOKEN · PROXMOX_PASSWORD"]
        S2["AUTHENTIK_TOKEN · AUTHENTIK_SECRET_KEY<br/>AUTHENTIK_POSTGRESQL_PASSWORD"]
        S3["NB_DATA_STORE_KEY · NB_RELAY_SECRET<br/>NB_PROXY_TOKEN · NB_BIFROST_SETUP_KEY"]
        S4["OPENBAO_UNSEAL_KEY"]
    end

    subgraph BOOT["Bootstrap — one-time laptop operations"]
        AGE["age private key<br/>~/.config/sops/age/keys.txt"]
        CS["just create-secrets<br/>bash scripts/create-bootstrap-secrets.sh"]
        KBS1["k8s Secret: openbao-unseal-key<br/>namespace: openbao"]
        KBS2["k8s Secret: cloudflare-api-token<br/>namespace: cert-manager"]
    end

    subgraph BIFROST["Bifrost VPS — auto-provisioned by Pulumi"]
        HG["hetzner.go<br/>bifrostEnvFiles"]
        SE["env/*.env on VPS<br/>one file per container, mode 0600"]
        DE["env/bootstrap.env on VPS<br/>NB_DATA_STORE_KEY · NB_RELAY_SECRET<br/>NB_OWNER_PASSWORD"]
        BS["bootstrap.sh<br/>starts services in order"]
        NC["sed substitution<br/>config.yaml ${VARS} → real values"]
    end

    subgraph RUNTIME["Runtime — OpenBao + CSI Driver"]
        OB["OpenBao<br/>openbao.madhan.app (KV v2)"]
        CSI["Secrets Store CSI Driver<br/>DaemonSet on all nodes"]
        SPC["SecretProviderClass per app<br/>defines: vault role, secret paths"]
        POD["Workload Pods<br/>files at /mnt/secrets<br/>or secretObjects → k8s Secret"]
    end

    AGE -->|decrypts| SOPSFILE
    SOPSFILE -->|sops exec-env| CS
    CS -->|kubectl create secret| KBS1
    CS -->|kubectl create secret| KBS2
    KBS1 -->|unseal sidecar reads key| OB

    SOPSFILE -->|sops exec-env| HG
    HG --> SE & DE
    SE & DE -->|CopyToRemote| BS
    DE --> NC
    NC -->|config ready| BS

    OB -->|K8s auth| CSI
    CSI -->|fetches secrets| SPC
    SPC -->|mounts files or syncs Secret| POD

Bootstrap Secrets (SOPS + age)

secrets/bootstrap.sops.yaml is age-encrypted and committed to git. It contains every secret needed to provision infrastructure. Decryption requires the age private key (~/.config/sops/age/keys.txt).

Populating the secrets file

# Open the encrypted file in $EDITOR — sops decrypts, you edit, it re-encrypts on save
sops secrets/bootstrap.sops.yaml

Required secrets (see Deployment Guide for full list):

# Infrastructure access
HCLOUD_TOKEN: <hetzner cloud API token>
PROXMOX_PASSWORD: <proxmox root password>
CLOUDFLARE_API_TOKEN: <zone-edit API token>

# Authentik — Bifrost VPS
AUTHENTIK_TOKEN: <60-char random string>        # bootstrap token → akadmin API token on first boot
AUTHENTIK_SECRET_KEY: <openssl rand -base64 48> # Django secret key
AUTHENTIK_POSTGRESQL_PASSWORD: <openssl rand -hex 16>
AUTHENTIK_GITHUB_SECRET: <github oauth app secret>

# NetBird — Bifrost VPS
NB_DATA_STORE_KEY: <openssl rand -base64 32>    # SQLite encryption key — never change after first deploy
NB_RELAY_SECRET: <openssl rand -base64 32>      # relay auth shared secret
NB_PROXY_TOKEN: <netbird personal access token> # added after first login to NetBird
NB_BIFROST_SETUP_KEY: <netbird setup key>       # added after first login to NetBird

# OpenBao — K8s secrets platform
OPENBAO_UNSEAL_KEY: <openssl rand -base64 32>   # generated once on first deploy

NB_DATA_STORE_KEY and NB_RELAY_SECRET must never change after the first deploy — the NetBird SQLite database is encrypted with these values. Rotating them means losing all peer registrations.

Bootstrap script

# Creates two k8s Secrets from the SOPS file:
just create-secrets
# → sops exec-env secrets/bootstrap.sops.yaml 'bash scripts/create-bootstrap-secrets.sh'
# → kubectl create secret generic openbao-unseal-key -n openbao
# → kubectl create secret generic cloudflare-api-token -n cert-manager

Both secrets carry argocd.argoproj.io/sync-options: Prune=false — Argo CD never deletes them.

Bifrost Auto-Provisioning

The Bifrost VPS secrets are fully automated by Pulumi. No manual SSH required.

just core hetzner up does the following in sequence:

1. bifrostEnvFiles — Pulumi reads SOPS env vars and writes one ./cloud/bifrost/env/<consumer>.env per container (plus bootstrap.env for the script) on the laptop, mode 0600
2. CopyToRemote uploads the entire ./cloud/bifrost/ directory to /etc/bifrost/ on the VPS
3. bootstrap.sh runs on the VPS:
   • Deletes the legacy shared .env/.secrets.env, then hardens the host
   • Starts all containers in dependency order, waiting for health at each step; each container reads only its own env file
   • Substitutes ${NB_RELAY_SECRET}, ${NB_DATA_STORE_KEY} and the bcrypt ${NB_OWNER_HASH} from env/bootstrap.env into netbird/config.yaml
   • Starts netbird-server (it has no env file; its secrets are in config.yaml)

See Hetzner Bifrost for the full bootstrap sequence.

Runtime Secrets (OpenBao + CSI Driver)

After the cluster is up, all application secrets are managed by OpenBao. Apps consume secrets via the Secrets Store CSI Driver — mounted as files in the pod, and optionally synced to a k8s Secret via secretObjects. The two patterns are described in Secrets.

Apps and their OpenBao paths

One-time setup

After first deploy, run just openbao-setup to configure K8s auth, policies, roles, and write initial secrets. See OpenBao for details.

What Lives Where — Quick Reference