Security

Where secrets live and how they move

Azure Key Vault holds application configuration; the cluster reads it through a CSI driver with its own identity. The parts that bite are the per-namespace copies nothing creates for you, and the order in which rotation has to happen.

Scope HQ productionVerified 8 October 2026Owner Infrastructure

1One vault per application

Each production application has its own Azure Key Vault, named dcbar-<app>-prod-kv, in tenant 89df4ea7-2a11-4867-8a15-e2e6102deb04. A SecretProviderClass in the application’s namespace lists which secrets to mount:

provider: azure
parameters:
  keyvaultName: dcbar-helpdesk-prod-kv
  tenantId: 89df4ea7-2a11-4867-8a15-e2e6102deb04
  usePodIdentity: "false"
  objects: |
    array:
      - |
        objectName: session-secret
        objectType: secret

The vault is created by the deployment generator’s setup commands. Its contents are always typed by a person — nothing populates a vault automatically.

A missing secret fails the whole mount

If one name in objects does not exist in the vault, the mount fails entirely and the pod never starts — and the message does not say which secret was absent. Compare the two lists directly:

az keyvault secret list --vault-name dcbar-helpdesk-prod-kv --query '[].name' -o tsv

2The identity that reads them

The Secrets Store CSI driver authenticates to Key Vault as an Azure service principal, created by the generator as app-kv-reader with a client secret valid two years and displayed once.

Whether that principal is shared across the estate or created per application is not currently documented, and it matters: if it is shared and someone runs the creation command again for a new application, they get a second principal with access to no vault at all, and the failure surfaces later as a pod stuck on a mount.

The cluster finds its credentials in a secret named secrets-store-creds, in the application’s own namespace. It is not cluster-wide and it is not copied for you:

kubectl -n <namespace> create secret generic secrets-store-creds \
  --from-literal=clientid="$SP_CLIENT_ID" \
  --from-literal=clientsecret="$SP_CLIENT_SECRET"

kubectl -n <namespace> label secret secrets-store-creds \
  secrets-store.csi.k8s.io/used=true

The label is required. Without it the driver ignores the secret and the pod sits in ContainerCreating reporting nothing more useful.

3Rotation, and why order matters

The driver is configured to refresh mounted files:

--enable-secret-rotation=true --rotation-poll-interval=2m

Files on disk update within two minutes of a vault change. A running process never re-reads them. So every rotation needs a restart as well — and the restart has to come second.

Restarting first produces a change that does nothing

This happened with the HelpDesk Entra client secret. The pods were restarted, read the old value into memory, and the CSI driver then updated the file underneath them. The pod hash changed, the deployment looked rolled, and the behaviour was identical — which reads as “the new secret is also wrong” rather than “the new secret was never loaded”. The correct order is always:

az keyvault secret set --vault-name <vault> --name <name> --value '<new>'
# wait for the poll interval, then
kubectl -n <namespace> rollout restart deploy/<app>

4Secrets that are not in a vault

Several things the estate depends on are held outside Key Vault, by hand. Each is one kubectl apply or one sync away from disappearing, and each disappearance is an outage.

What	Where it lives	How it got there
Wildcard TLS certificate	A `production-<app>` secret in every namespace that serves a hostname	Copied by hand from an existing namespace
GHCR pull credentials	`ghcr-pull-secret` per namespace	Created by hand from a personal access token
CSI credentials	`secrets-store-creds` per namespace	Created by hand
SMTP credentials for alerting	`watchdog-smtp` in `database`	Created by hand
Cloudflare tunnel routes	The Cloudflare dashboard	Not in Git at all

The wildcard expires 27 November 2026

*.dcbar.org, issued 20 November 2025. Every production hostname depends on it, and because each namespace holds its own copy, renewal means replacing it in every one of them. The list of namespaces holding a copy is itself something that has to be kept current — there is no single place that records it.