AIM GOLD — backup and recovery¶

What is backed up, what is not, and exactly how to get the system back.

Every statement here is about the implementation in this repository. Where something is not yet protected, it says so plainly.

1. Recovery inventory¶

The question this table answers is not "what do we copy" but "what does a brand new machine need before AIM GOLD is running again".

#	Item	Where it lives	Classification	Notes
A	PostgreSQL database	`digigold_pg_data` volume	BACKED_UP_ENCRYPTED	`pg_dump` → verify → encrypt → offsite. The offsite half is inert until a destination is configured.
B	Object storage — KYC documents, KYC selfies, invoice PDFs, product images	`digigold_minio_data` volume	BACKED_UP_ENCRYPTED	New in Phase 6H. Mirrored via `mc`, tarred, then through the same encrypt/upload/verify path.
C	`FIELD_ENCRYPTION_KEY`	`.env.production` on the host, one copy	SEPARATE_SECRET_RECOVERY_REQUIRED	Decrypts `kyc_records.pan_encrypted`, `kyc_records.aadhaar_encrypted`, `bank_accounts.account_no_enc`. Deliberately not in the backup. See §5.
D	`BACKUP_ENCRYPTION_KEY`	owner's password manager	SEPARATE_SECRET_RECOVERY_REQUIRED	Opens the offsite archives. Must never be stored with them. See §5.
E	Gateway/provider credentials — Razorpay, SafeGold, SMS	`.env.production`	EXTERNALLY_MANAGED	Re-obtainable from each vendor's dashboard. Losing them is an inconvenience, not data loss.
F	`JWT_SECRET`	`.env.production`	RECREATABLE	Regenerating it invalidates live sessions; customers sign in again. No data is lost.
G	Database/Redis/MinIO passwords	`.env.production`	RECREATABLE	Set them to new values when rebuilding; they only have to match themselves.
H	Application code, migrations, deployment config	git	RECREATABLE	`git clone` and deploy.
I	TLS certificates	`caddy_data` volume	RECREATABLE	Caddy re-issues from Let's Encrypt on first boot.
J	Redis contents	`digigold_redis_data`	RECREATABLE	Sessions, OTP state, caches, the reconciler's due-time. All rebuild.
K	Prometheus / Grafana / Loki / Tempo data	monitoring volumes	RECREATABLE	Telemetry history. Its loss costs graphs, not money.
L	Accounting policy determinations and statutory parameters	PostgreSQL	BACKED_UP_ENCRYPTED	Inside the database dump; the append-only history comes back with it.

The two that decide whether a recovery succeeds are C and D, and neither is in any backup. That is deliberate — a key stored inside the thing it encrypts protects nothing — and it is why §5 exists.

2. What a backup run does¶

Daily at 02:30 UTC (08:00 IST), and on every deploy and migration.

  pg_dump ──► .partial ──► verify ──► promote ──► local archive
                            │
                            ├─ non-empty
                            ├─ ≥ 10 KiB
                            ├─ gzip integrity
                            └─ "PostgreSQL database dump complete" marker

  mc mirror ─► tar.gz ────► verify ──► promote ──► object archive
                            │
                            ├─ non-empty
                            ├─ gzip integrity
                            └─ tar listing readable

  each archive ─► encrypt ─► checksum ─► upload ─► READ BACK ─► compare ─► DECRYPT ─► retain

The last four steps are the ones that matter. An upload that returns zero and stored nothing is indistinguishable from a working backup until the day it is needed, so the remote copy is downloaded again, checksum-compared against the local ciphertext, and actually decrypted before anything is called verified.

Only then does retention run, and it will not touch the newest archive or the one just uploaded.

Safety properties¶

The dump is created under umask 077, so it is never world-readable — not even during the window in which it is being written.
.partial staging means a half-written file can never be mistaken for a backup.
flock /tmp/aim-deploy.lock is shared with every deploy and migration, so a backup can never run concurrently with one.
A failed upload never deletes the local archive.
A failed object backup never invalidates the database archive.

3. Encryption¶

openssl enc -aes-256-cbc -md sha512 -pbkdf2 -iter 600000 -salt \
  -pass env:BACKUP_ENCRYPTION_KEY

AES-256-CBC with a random salt per archive.
PBKDF2, 600,000 iterations, SHA-512. The passphrase is human-supplied and the ciphertext will sit in third-party storage where an attacker has unlimited offline attempts; without a strong KDF, openssl's legacy derivation is a single MD5 pass.
The key is passed by environment variable, never as an argument — arguments are visible in ps to every user on the box.
The key is never logged, never echoed, and never part of a filename.
Keys shorter than 32 characters are refused, as are unreplaced placeholders.

Which key opens which archive¶

The key id is in the filename:

scheduled-20260812-064152.k1.sql.gz.enc
                          ▲
                          └── encrypted with the key registered as k1

Once encrypted, a file cannot say what opens it. An operator three rotations later has no other way to tell, which is why this is in the name rather than in a database somewhere that may itself be gone.

4. Recovering from total VPS loss¶

The scenario every other procedure assumes away: the host is destroyed, seized, or irrecoverable. You have only the repository, the offsite archives, and the keys you stored elsewhere.

Provision a new host and install Docker.
Clone the repository. Everything in deploy/ is there.
Fetch the newest offsite archives — one scheduled-*.sql.gz.enc and one objects-*.tar.gz.enc. Note the .k?. segment in each name.

Check they open before you rely on them:

openssl enc -d -aes-256-cbc -md sha512 -pbkdf2 -iter 600000 \
  -in scheduled-<stamp>.k1.sql.gz.enc \
  -pass env:BACKUP_ENCRYPTION_KEY | gzip -t && echo "archive is intact"

Generate a fresh environment: bash deploy/gen-env.sh, then fill in:
FIELD_ENCRYPTION_KEY — the original value, from your key store. A new one will not decrypt existing PAN, Aadhaar or bank-account columns.
provider credentials (Razorpay, SafeGold, SMS) from each vendor's console.
fresh database/Redis/MinIO passwords — these only need to match themselves.

Start Postgres and MinIO, then restore:

openssl enc -d … -in scheduled-<stamp>.k1.sql.gz.enc … | zcat \
  | docker exec -i digigold_postgres psql -U "$DB_USER" -d "$DB_NAME" -v ON_ERROR_STOP=1

openssl enc -d … -in objects-<stamp>.k1.tar.gz.enc … | tar -xzf - -C /tmp/objects
docker run --rm --network aim_data -v /tmp/objects:/restore \
  -e MC_HOST_local="http://$S3_ACCESS_KEY:$S3_SECRET_KEY@digigold_minio:9000" \
  minio/mc mirror --preserve /restore local

Bring the stack up and verify with production-db-sanity: migration names must match the repository, and ledger_entries must match what the backup contained.

Every step above uses only the three inputs named at the start. Nothing requires a file that existed solely on the destroyed host — which is precisely what makes items C and D in the inventory non-negotiable.

5. The keys, and what happens if you lose them¶

`BACKUP_ENCRYPTION_KEY`¶

If this is lost, every encrypted archive is permanently unreadable. There is no recovery, no vendor to call, and no support path. AES-256 with a 600,000-iteration PBKDF2 is not brute-forceable.

Rules:

Never store it in the backup destination. An attacker who obtains the bucket would then hold both the ciphertext and the key, and the encryption would have achieved nothing.
Never store it only on the VPS. A host loss would take the key and the data together, which is the exact scenario this system exists to survive.
Never commit it, log it, or pass it as a command argument.
Keep it in a password manager, plus one offline copy somewhere physically separate.

`FIELD_ENCRYPTION_KEY`¶

Different key, different job, same severity. It decrypts three columns: kyc_records.pan_encrypted, kyc_records.aadhaar_encrypted, and bank_accounts.account_no_enc.

It is deliberately excluded from the database dump — including it would put the key inside the archive it protects. So a restore performed without it yields a database that is structurally perfect and in which full PAN, full Aadhaar and full bank account numbers are permanently undecryptable. Only the _last4 columns survive, payout dispatch stops working, and the affected customers must re-submit their identity documents.

Store it exactly as you store the backup key, and separately from it.

Rotating the backup key¶

Generate the new key: openssl rand -base64 48. Keep the old one — every archive already written still needs it.
Set BACKUP_ENCRYPTION_KEY and bump BACKUP_ENCRYPTION_KEY_ID (k2, k3, …) in /opt/aim/deploy/.env.production.
Run scheduled-backup once and confirm it reports OFFSITE_BACKUP_VERIFIED. That step decrypts the uploaded copy with the new key before reporting success, so a rotation producing unopenable archives fails at rotation time rather than during an incident.
Record the changeover: key id, date, and the archives it covers.
Retire the old key only once every archive encrypted under it has aged out.

6. Retention¶

	Local	Offsite
Kept	14 per family (`scheduled-`, `pre-deploy-`, `pre-migrate-*`)	30
Ordering	`ls -t`, newest first	by the timestamp embedded in the filename
Newest protected	structurally — `tail -n +15` can never select it	explicitly, plus the archive just uploaded
Floor	—	refuses to report success if the destination ends up empty

The two policies are independent. They protect against different failures — a bad deploy versus a lost host — and a single shared rule would let one erode the other.

Offsite ordering is by embedded timestamp rather than filename, because pre-migrate-20260812 sorts before scheduled-20260811 lexicographically: a newer archive would have been selected as the oldest and deleted.

Set OFFSITE_RETENTION_DRY_RUN=true to see what would be deleted without deleting anything.

7. Checking the state without an alert transport¶

No alert delivery is configured, by choice. Two surfaces report the truth:

bash /opt/aim/deploy/scripts/backup-status.sh show

and the BACKUP STATUS section of the production-db-sanity workflow, which needs no SSH access.

Both report two separate verdicts, and the distinction is the point:

RESULT: LOCAL_BACKUP_OK — verified 0h 12m ago
RECOVERABLE OFF-HOST: NO — off-host storage is switched off by owner
  decision, so a lost host loses the database and every archive.
RESULT: OFFSITE_BACKUP_NOT_CONFIGURED

A local archive is not a recovery point for host loss, and this surface refuses to describe it as one. It also refuses to report the absence as a failure when it is a decision — see §8.

The same facts are published as Prometheus metrics through node_exporter's textfile collector, so time() - aim_backup_local_last_success_timestamp is a real query whenever someone wants an alert rule. Timestamps and counts only — no filenames, no paths, no labels.

8. The four off-host states, and how to change them¶

offsite-backup.sh ends every run with exactly one STATE: line. Four states, because three would force two different situations to share a word:

State	Exit	Means
`OFFSITE_BACKUP_NOT_CONFIGURED`	0	The owner has decided there is no off-host copy.
`OFFSITE_BACKUP_OK`	0	A remote copy exists, was read back, digest-matched and decrypted.
`OFFSITE_BACKUP_FAILED`	2	An off-host copy was expected and did not happen.
`OFFSITE_BACKUP_UNDECLARED`	3	Nobody has said whether one is expected. A configuration defect.

NOT_CONFIGURED and OK both exit 0 and mean opposite things. Only one of them survives losing this host. The state line is what tells them apart, because an exit code cannot.

Why `UNDECLARED` fails¶

An unset OFFSITE_BACKUP_REMOTE used to print "destination required" and exit 0. So a deliberate decision and a broken configuration produced the same green run — and the broken configuration was not hypothetical. PR #75 shipped . ./.env.production instead of set -a; . ./.env.production, which meant a fully configured destination was invisible to the backup script. It reported "destination required", exited 0, and the scheduled run went green with no off-host copy and nothing to say why.

Silence is therefore no longer a valid state. Intent must be declared.

Where the decision lives¶

deploy/backup-policy.env — committed, non-secret, attributable, and shipped to the host by the backend deploy. It is a policy, not a credential: it belongs in version control the same way production-migration-authorization.json does.

Current value: OFFSITE_BACKUP_MODE=disabled. Third-party object storage was evaluated and declined by the owner on 2026-08-13. The residual risk is stated plainly and accepted: a total loss of the VPS loses the database and every archive together.

Turning off-host backup on later¶

No change to this repository is required, and no code change of any kind. In /opt/aim/deploy/.env.production:

OFFSITE_BACKUP_REMOTE=<rclone remote>:<bucket>
BACKUP_ENCRYPTION_KEY=<from `openssl rand -base64 48`, stored off this host>
BACKUP_ENCRYPTION_KEY_ID=k1

A configured destination outranks the disabled policy, deliberately: a stale line in a policy file must never be able to silently skip uploads somebody had configured and was relying on.

The transport is rclone, so the destination is provider-neutral — any S3-compatible store, Backblaze B2, an SFTP host, or a second machine you own. Nothing in this repository names or requires a particular vendor. rclone must be installed on the host; the script reports OFFSITE_TOOLING_MISSING if it is not.

9. The host-local backup timer (prepared, NOT enabled)¶

The problem it solves¶

Every backup path reached this host over SSH from a GitHub runner. Phase 6H watched two consecutive scheduled runs fail at dial tcp <vps>:22: i/o timeout — SSH never connected, so no backup logic ran at all — while production was completely healthy and answered 200 on every endpoint.

That is the worst shape a failure can have: backups stop, every customer-facing indicator stays green, and with no alert transport configured the only signal is a red job in a tab nobody is required to open.

The backup logic now lives in deploy/scripts/local-backup.sh, and both schedulers run it:

.github/workflows/scheduled-backup.yml — GitHub, 02:30 UTC, over SSH
deploy/systemd/aim-backup.timer — the host, 03:30 UTC, no network at all

The timer is a fallback, not a second daily backup. It passes BACKUP_MIN_INTERVAL_SECONDS=75600 (21 hours), so on a normal day it finds the archive GitHub already made and stands down without dumping anything. It only takes a backup on the days the primary schedule did not.

Both serialise on flock /tmp/aim-deploy.lock, the same lock every deploy and migration takes, so a backup can never run concurrently with either — or with itself.

Enabling it¶

This is an operator action and is deliberately not performed by any workflow. A deploy that silently changed the production backup schedule would be a change nobody authorised. The unit files are shipped to /opt/aim/deploy/systemd/ by deploy-backend; enabling them is two commands.

sudo install -m 0644 /opt/aim/deploy/systemd/aim-backup.service /etc/systemd/system/
sudo install -m 0644 /opt/aim/deploy/systemd/aim-backup.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now aim-backup.timer

Verify without waiting a day:

systemctl list-timers aim-backup.timer     # NEXT/LEFT columns show the schedule
sudo systemctl start aim-backup.service    # run it once, now
journalctl -u aim-backup.service -n 50     # read what it did

The one-off run will normally print LOCAL_BACKUP_SKIPPED_RECENT if a backup was taken in the last 21 hours — that is the fallback logic working, not a failure. To prove the full path instead:

sudo systemd-run --unit=aim-backup-test --wait --property=Environment=BACKUP_MIN_INTERVAL_SECONDS=0 \
  /usr/bin/flock --wait 900 /tmp/aim-deploy.lock /bin/bash /opt/aim/deploy/scripts/local-backup.sh

To disable: sudo systemctl disable --now aim-backup.timer.

Neither unit contains a secret. local-backup.sh reads .env.production itself at run time, as root; the unit files are world-readable and there is no EnvironmentFile=.

10. Making an offline recovery copy of `FIELD_ENCRYPTION_KEY`¶

Read §5 first. This key decrypts kyc_records.pan_encrypted, kyc_records.aadhaar_encrypted and bank_accounts.account_no_enc. It exists in exactly one place — /opt/aim/deploy/.env.production on the VPS — and it is deliberately not in any backup, because a key stored inside the thing it locks protects nothing.

If the host is lost and no copy of this key exists, those three columns are permanently unrecoverable even from a perfect database backup. Everything else restores. Those do not. Only the _last4 columns survive, payout dispatch stops working, and every affected customer must re-submit their identity documents.

With off-host backup switched off (§8), the whole database is already lost with the host — so today this key is the second thing that would be lost, not the first. It still matters: the decision about off-host storage can be revisited at any time, and a copy of this key made now is one that already exists when it does.

Rules¶

Never commit it, never paste it into a chat window (including an AI assistant), never put it in a GitHub secret used by a workflow that echoes, never store it in the same place as a database backup.
Never rotate it. It decrypts data already written; a new key does not.
The value is 64 hex characters. Nothing else is the key.

Procedure — owner only, one sitting¶

This runs from your own terminal over SSH, never through a GitHub workflow: a workflow writes its output to a run log that GitHub retains and that anyone with repository access can read.

1. Open a session that does not record what you type.

ssh <user>@200.141.2.55
unset HISTFILE

unset HISTFILE stops this shell writing ~/.bash_history on exit. It protects the commands; step 4 handles the output.

2. Take the fingerprint first — this is safe to write down anywhere.

sudo grep -m1 '^FIELD_ENCRYPTION_KEY=' /opt/aim/deploy/.env.production \
  | cut -d= -f2- | tr -d '\r\n' | sha256sum

This prints a 64-character SHA-256 of the key. It is not the key and cannot be turned back into it. Record it in your key register, and in a comment in your password-manager entry. It is what lets you prove later that a stored copy is correct without ever displaying the key again.

3. Display the key once, and copy it straight into your password manager.

sudo grep -m1 '^FIELD_ENCRYPTION_KEY=' /opt/aim/deploy/.env.production | cut -d= -f2-

Create an entry named AIM GOLD FIELD_ENCRYPTION_KEY (production), paste the value, and paste the fingerprint from step 2 into its notes.

Then make the second copy — printed and stored physically, or a second password manager. One copy in one system is not a recovery plan.

4. Clear the screen and close the session.

clear && printf '\033[3J'
exit

The second sequence clears the scrollback buffer, which clear alone does not. If your terminal keeps its own log, delete that log too.

5. Verify your stored copy, without displaying the key again.

On your own machine, paste the value from your password manager into:

read -rs KEYCHECK && printf '%s' "$KEYCHECK" | sha256sum && unset KEYCHECK

read -rs does not echo and does not enter history. Compare the output with the fingerprint from step 2. If they match, you have a working recovery copy. If they do not, the stored copy is wrong — most often a truncated paste or a trailing space — and it would be discovered during an incident.

Record the date you completed this in your key register. The fingerprint is safe to include in an ops report; the key never is.