Backups and upgrades#

What to back up, how to upgrade, and what a rollback will not restore.

What holds state#

Back up#

1. Back up Postgres with your normal tooling, to a known-consistent point.
2. Back up /var/lib/midnight, including the generated keys.
3. Back up /etc/midnight.
4. Record the package version installed on the host.
5. Store the key backup with the protection you would give a private key — separately from the database backup, encrypted.

Taking the database and the keys as one logical set is what makes the backup restorable.

Restore#

1. Restore Postgres.
2. Restore /var/lib/midnight and /etc/midnight.
3. Install the same package version you recorded.
4. Start the units and read the doctor report.
5. Verify the audit ledger. See Monitoring and audit.

Upgrade#

1. Back up first, as above.
2. Read the release notes for migrations.
3. Install the new package. It runs available migrations when a connection string is configured, and writes a fresh doctor report.
4. Read the doctor report before enabling or restarting units.
5. Verify the ledger and deploy a low-risk service to confirm the path works end to end.

What rollback does not cover#

midnight rollback --previous returns a deployment. It does not:

• reverse a database migration your application ran;
• revert variables you changed; or
• restore control-plane state.

Decide what else needs reversing before relying on it. See Deploying and rolling back.

Key rotation#

The variable vault supports rotating the key-encryption key, and keeps immutable encrypted version history. Rotation is an operation to plan and back up around, not a routine command.

Verify#

• A restored installation starts with all six units active.
• A sealed variable can still be used by a deployment, proving the keys match the database.
• midnight audit verify succeeds.

If it does not work#