Renewals and expiry reminders

1What this replaces

Nothing. Until now no system has watched any expiry date in this estate. The wildcard certificate, the Azure service principal secret, the GitHub tokens and the SendGrid key all expire on fixed dates, and the mechanism for noticing was somebody remembering.

Each of those failures is quiet. An expired certificate breaks every production hostname at once; an expired service principal secret stops pods mounting their configuration, so they fail to start on their next restart rather than immediately; an expired registry token stops image updates and says nothing at all. None of them announces itself in advance.

2How it works

A CronJob in namespace ops reads a CSV register and emails through the same SendGrid path the database watchdog uses. It runs daily but sends only on threshold days — 90, 60, 30, 14, 7, 3 and 1 days before, and on the day itself:

Anything already past its date sends every day until the register is corrected, and exits non-zero so the Job also shows as failed in kubectl. An expired item should be impossible to ignore quietly.

A malformed row is reported in the same way. That matters more than it sounds: silence from this job has to mean “nothing is due”, never “the file could not be read” — which is exactly how the database watchdog managed to fail for six weeks without anyone noticing.

3The register

One row per dated item, in deploy/renewals.yaml:

date,item,category,owner,note

It currently contains one entry:

To add something, edit the register and commit. There is no interface and no database — the history of who added what, and when, is the Git log.

4The items with no date

These have hard expiries and nobody has recorded when. They cannot be added to the register until someone goes and looks, and that is the single most useful piece of work this page points at.

5Deploying it

The job needs SMTP credentials in its own namespace. They are the same ones the database watchdog uses, so copy rather than recreate:

kubectl apply -f deploy/renewals.yaml

kubectl -n database get secret watchdog-smtp -o yaml \
  | sed -e 's/namespace: database/namespace: ops/' \
        -e 's/name: watchdog-smtp/name: reminders-smtp/' \
        -e '/resourceVersion\|uid\|creationTimestamp\|selfLink/d' \
  | kubectl apply -f -

Then prove it end to end rather than waiting a day to find out:

kubectl -n ops create job --from=cronjob/renewals renewals-test-$(date +%s)
kubectl -n ops logs job/renewals-test-<suffix>

With only the certificate in the register and more than 90 days to go, the expected output is nothing due and no email. To prove the mail path, add a row dated today, run it again, and check the channel — then remove the row.