Skip to content

0015 — Production migrations are an armed, single-use act

Status: accepted

Context

Applying a migration used to be a line inside deploy-backend.yml. Merging a pull request was therefore the act that altered the production schema. Nobody chose that; it was a side effect of shipping code, and it happened at whatever hour the merge landed. On 2026-08-07 a merge did exactly that, and the record of it (221188c) is titled "docs: record that merging applies the migration to production".

Two problems compound. migration:run applies every pending migration in one transaction, not a selected one — so the blast radius of a merge was whatever had accumulated. And because deploy and migrate were the same step, there was no way to order them safely: an additive migration could not be applied and verified before the code that depends on it shipped.

The obvious control is four-eyes approval. It is not available here. This repository is operated by one GitHub account (aimgoldorg) by explicit choice of its owner: no second approver, no organization, no paid plan. GitHub Environments were configured as far as the plan allows (8c0523a), and required reviewers were refused by the API — so environment: production approves instantly and silently, which is worse than no gate because the file looks protected.

Decision

Split migration out into a dispatch-only migrate-production.yml, and gate it on an armed manifest committed to the repository: deploy/production-migration-authorization.json. Editing that file is the authorizing act. aim-digigold-real/scripts/verify-migration-authorization.js enforces it, and refuses unless every binding holds at the moment of dispatch:

Binding What it pins
Commit-bound approved_commit_sha must match the running commit, and only the manifest file itself may differ from it. Any other moved file rejects the run.
List-bound authorized_migrations must name the pending migrations exactly, and in order.
Operator-bound approved_by must equal the GitHub login triggering the run. An authorization armed for one account is inert to another.
Single-use authorization_id must not appear in /opt/aim/deploy/consumed-migration-authorizations.txt, a server-side ledger of spent ids.

Plus: the ref must be refs/heads/main, expires_at must not have passed, the dispatcher must type confirm=MIGRATE (dry_run defaults to true), and a verified pre-migrate-* backup must exist. The workflow shares a concurrency group with deploy-backend, so a migration and a deploy can never be in flight at the same time — with cancel-in-progress: false on both, the second queues and waits rather than being cancelled.

Because deploy no longer migrates, an image could otherwise reach production expecting a schema that is not there. deploy-backend.yml therefore refuses to deploy while any migration is pending — the failure is a stopped deploy rather than a runtime error.

This is SINGLE_OWNER_CONTROL_MODE, and it is not four-eyes approval. An earlier revision required approved_by !== actor, forcing a second identity to spend what a first had armed. That requirement was removed deliberately, not weakened by accident, because the second identity does not exist. What the gate buys is that a production migration cannot happen by merging code, by accident, or by typing one word: it takes a deliberate commit naming an exact code state and an exact ordered list, a manual dispatch, a typed confirmation, an unexpired one-time id, and a verified backup. What it does not buy is independent review. One mistake, or one compromised account, is sufficient. Only the first property is claimed. See AIM_GOLD_PRODUCTION_MIGRATION_GOVERNANCE.md.

Consequences

  • A release carrying a migration is three steps, and the middle one is mandatory: run deploy-backend (it builds and tags the image, then refuses), run migrate-production, then run deploy-backend again.
  • On the manual path, the migration image tag comes from approved_commit_sha, never HEAD. Arming is itself a commit, so HEAD is always one ahead of the built image and a HEAD-derived tag names an image that never existed. That was a real bug in deploy/scripts/manual-deploy.sh, fixed in 6bf91a8 — and found only because the authorization then had to be re-armed against the fix. The Actions workflow has no equivalent handling: it migrates against whatever API_IMAGE_TAG the server's .env.production pins.
  • Disarming is part of the procedure. authorization_id: null is the resting state. A spent authorization left armed is a misleading artefact even though the consumed ledger already blocks it.
  • TypeORM only sees migrations compiled into the image it runs from. migration:show against a stale :latest will confidently report "none pending" for a migration that exists in git. manual-deploy.sh overrides API_IMAGE_TAG for that one invocation, because it promotes :latest only later; deploy-backend.yml needs no override, because it pulls the freshly built :latest before asking the question.
  • The manifest is repository-visible and must never contain a secret. SHAs, migration names and logins are not secrets; passwords and connection strings are.
  • Rollback stays manual — typeorm migration:revert, or restore from /opt/aim/backups/pre-migrate-*.sql.gz. An ordinary deploy can never evict a pre-migrate-* archive: deploy-backend.yml prunes only pre-deploy-*. The migrate workflow's own retention is broader — it keeps the newest 14 of pre-*, both families competing for the same slots.
  • Writing the gate surfaced two bugs in itself before it was trusted: the pending list was parsed from "everything before" rather than a bounded region (fd56a9a), and the stdout the gate depends on was not captured (7bad0e6).