Failing over to AWS

Phase 1 — Decide, then stop the primary

Everything here is reversible. Nothing after Phase 2 is.

kubectl --context kubernetes-admin@kubernetes get nodes
curl -s -o /dev/null -w 'http=%{http_code} verify=%{ssl_verify_result}\n' https://prod-argocd.dcbar.org/
ping 10.20.100.227   # the API-server VIP

kubectl --context kubernetes-admin@kubernetes -n database \
  patch cluster app-db --type merge -p '{"spec":{"instances":0}}'

Phase 2 — Promote the database

The point of no return. After this, HQ is behind and stays behind.

kubectl --context aws -n database exec app-db-1 -c postgres -- psql -X -c \
  "SELECT pg_is_in_recovery() AS replica,
          pg_last_xact_replay_timestamp() AS last_replayed,
          now() - pg_last_xact_replay_timestamp() AS lag;"

kubectl --context aws -n database \
  patch cluster app-db --type merge -p '{"spec":{"replica":{"enabled":false}}}'

kubectl --context aws -n database exec app-db-1 -c postgres -- \
  psql -d nomination-portal-prod -X -c \
  "CREATE TABLE IF NOT EXISTS failover_marker(at timestamptz default now());
   INSERT INTO failover_marker DEFAULT VALUES;
   SELECT pg_is_in_recovery(), count(*) FROM failover_marker;"

Phase 3 — Bring applications up

Written for the world where the ApplicationSet exists. Today this phase is done by hand.

kubectl --context aws get deploy -A

prod-db.dcbar.org  →  a710caeabde924bd78dd0b5cf5767523-f05aa628024352eb.elb.us-east-1.amazonaws.com
                       (currently 10.40.100.162, but use the name — NLB addresses change)

kubectl --context aws -n <app-namespace> rollout restart deploy --all

psql "host=prod-db.dcbar.org hostaddr=10.40.100.162 \
      sslmode=verify-full sslrootcert=system \
      dbname=nomination-portal-prod user=dbadmin" \
  -c 'SELECT pg_is_in_recovery(), count(*) FROM submissions'

Phase 4 — Move the traffic

The cutover itself. Fast, and reversible in one click.

Phase 5 — Verify, then watch

curl -sI -H 'Host: hello-world-tim.dcbar.org' http://10.40.100.222/ | head -3

kubectl --context aws -n traefik port-forward deploy/traefik 9000:8080
# then, in another shell
curl -s localhost:9000/api/http/routers | python3 -c "
import sys, json
for r in json.load(sys.stdin):
    print('%-55s -> %s' % (r.get('rule',''), r.get('service','')))
"

curl -sS -o /dev/null -w '%{http_code}\n' https://<your-public-hostname>/

kubectl --context aws -n database set env cronjob/db-watchdog MODE=hq
kubectl --context aws -n database patch cronjob pg-heartbeat \
  -p '{"spec":{"suspend":false}}'

Failing back costs a re-seed

Plan for this before you need it, not during.

The moment AWS accepts one write, HQ is stale and cannot resume as primary — its timeline has diverged. Failing back means rebuilding HQ as a replica of AWS, waiting for it to catch up, and then performing this same procedure in reverse, deliberately, during a quiet window.

Do not attempt to restart the old HQ primary and let the two reconcile. They will not.

What has to be built before this is a real plan

Each of these turns a manual step above into an automatic one.

• An ApplicationSet covering both clustersOne definition deploying to HQ and AWS, with AWS at zero replicas. Failover becomes a replica count, not a hand-built deployment written under pressure.
• A decision on Argo CD at AWSA second instance watching the same repo, so a HQ outage does not remove your deployment tooling at the moment you need it most. Or accept manual kubectl and say so here.
• A rehearsalPromote, confirm writes, rebuild as a replica — on a Tuesday afternoon. Every number in this document is an estimate until someone has done it once.
• Secrets that live in Git, not in two clusters by handThe GHCR pull secret, the wildcard certificate and the watchdog credentials were all copied across manually. None survives a namespace rebuild, and none exists for a namespace Argo CD creates next. Sealed Secrets or External Secrets makes them part of the deployment rather than folklore.
• An RTO everyone agrees onCurrently unknown, because the procedure has never been timed end to end. The database half is now proven; the application half is not. That number decides whether the gaps above are worth closing.
• A 60-second TTL on prod-db.dcbar.orgStill on Auto. The cutover is a DNS edit, so the TTL is the floor on how long clients keep talking to a dead site. Lower it before you need it, not during.
• Live patches committed to GitFixes applied with kubectl patch are reverted on the next Argo CD sync, producing new ReplicaSets at unpredictable moments — each one re-rolling the dice on which node a pod lands on, so the same fault reappears looking like a different problem. Also still in Git: the leading space in destination.namespace.

Traps already discovered

Each of these cost real time to find. None produced an obvious error message.

• CloudNativePG restarts. The annotation is kubectl.kubernetes.io/restartedAt. An unrecognised one is ignored silently — the cluster never restarts, and every downstream check still passes.
• pg_switch_wal() on an idle database. A no-op. It will not ship a segment if nothing has been written, so a "force an archive" test proves nothing without a write first.
• archive_timeout without traffic. Also does nothing. The 5-minute heartbeat exists purely so the timer has something to flush, which is what bounds RPO.
• Barman and serverName. Backups are organised by server name. Omit it in externalClusters and a restore looks in an empty catalog: no target backup found.
• Two tunnels, one token. Using the same Cloudflare token at both sites registers both as connectors on one tunnel, and traffic round-robins between them.
• Traefik's Service type. The Helm chart reads service.spec.type, not service.type. The wrong key is accepted, stored in the release values where it looks correct, and read by nothing — leaving a LoadBalancer, which on EKS means an ELB nobody asked for.
• Traefik's internal port 8000. The web entrypoint already binds it inside the pod. A second entrypoint on the same container port crash-loops.
• Exported AWS_* variables. They hijack aws eks get-token and turn a working kubeconfig into Unauthorized.
• Each cluster used to have its own database CA — resolved 2 Sep 2026. CloudNativePG signs with a CA it generates per cluster, and certificates are not in WAL, so HQ and AWS had different authorities. Every application had to mount a CA bundle, and promotion would have failed with SELF_SIGNED_CERT_IN_CHAIN — a TLS error at the worst moment with nothing wrong with the data. Both clusters now serve the public *.dcbar.org certificate instead, so no client carries a CA at all. If a cluster is ever rebuilt, re-apply db-server-tls / db-server-ca and the certificates block, or it silently reverts to its own CA.
• serverAltDNSNames and serverTLSSecret are mutually exclusive. CloudNativePG only adds names to certificates it issues itself, so supplying your own is rejected until the alt-names list is removed in the same patch ("serverAltDNSNames":null).
• The pooler needs a restart for a new certificate. PgBouncer reads the server certificate from a mounted secret at startup, so the instances serve the new one while the pooler — which is what clients actually reach — keeps serving the old one. It looks like the patch failed.
• libpq ignores the system trust store. sslmode=verify-full alone looks for ~/.postgresql/root.crt and fails when it is absent, even for a publicly trusted certificate. Add sslrootcert=system (Postgres 16+).
• loadBalancerSourceRanges must include the pod network. HQ pods live on 10.244.0.0/16. Omit it and a pod scheduled onto the node that owns the MetalLB VIP hairpins into kube-proxy's reject rule and gets EHOSTUNREACH, while pods on every other node work normally via the service short-circuit. It presents as one replica of three crash-looping, and it moves whenever the VIP moves.
• A Cloudflare DNS record decides which tunnel serves a hostname. The tunnel's Routes page only says what that tunnel would serve. A record pointing at dcbar-hq sends the request to HQ no matter how correct the AWS routes look — verify with the DNS tab, or the tunnel's Live logs, which stay empty if the request never arrives.
• An archive-based replica never appears in pg_stat_replication. It replays WAL from S3 rather than streaming, so the primary has no connection to report. An empty replication view is expected here, not evidence that replication is broken — check pg_last_xact_replay_timestamp() on the replica instead.
• Pod addresses come from the node's subnet. EKS gives each pod an address from whichever subnet its node sits in, so a VPN that carries only some of the VPC's subnets makes HQ reachable from some pods and not others — decided by the scheduler, and changing on every restart. It presents as an intermittent timeout with no pattern. Check every subnet the node group can use, not just the one you tested from.
• AWS load balancers land where they like. Left alone, an NLB spreads across every subnet in the VPC — and the VPN only carries 10.40.100.0/24, so five of its six addresses were unreachable. Pin it with aws-load-balancer-subnets rather than widening the tunnel.
• EKS LoadBalancer scheme. The in-tree provider reads aws-load-balancer-internal: "true". The newer aws-load-balancer-scheme annotation belongs to the AWS Load Balancer Controller, which eksctl does not install — so it is ignored, and the default is internet-facing. Always set loadBalancerSourceRanges too, and check the address you get back is a VPC one.
• keepalived weights. A positive weight only adds a bonus while healthy. To move a VIP on failure it must be negative.

Reference