Monitoring and audit#

Check that an installation is healthy, and verify that its audit record has not been tampered with.

The doctor report#

The package writes a report at install time:

cat /var/lib/midnight/doctor-postinst.json

Read it before enabling units, and again after every upgrade. It is the fastest statement of whether the host meets what Midnight needs.

A midnight-doctor binary ships alongside the server for the same purpose outside install time.

Unit health#

systemctl status midnight-server midnight-worker midnight-agent

journalctl -u midnight-worker -n 200

Six units make up an installation: server, worker, agent, buildkit, buildkit-worker and registry. A deploy that queues and never builds usually means worker or buildkit is not running.

The audit ledger#

Midnight keeps a tamper-evident audit log: a SHA-256 hash chain where each row's hash covers the previous hash and the canonical row, appended INSERT-only under an advisory lock, with signed checkpoints.

Verify it:

midnight audit verify

Important

Verification proves the chain is internally consistent — that no row was altered or removed without breaking it. It does not prove that everything which happened was recorded.

Run it after a restore, after an upgrade, and whenever you have reason to doubt the record. The ledger also supports export, so a verified copy can be kept off the host.

What is audited#

Agent tool calls are audited per call, including mutating control-plane actions. Variable decrypt operations are audited with execution attribution and redaction before production build or runtime injection — so a decrypt is recorded without the value being recorded.

Deployment state#

The canonical deployments table has a database-level guard trigger enforcing legal state transitions. A state you read is a state the system permitted, which makes deployment records dependable during an incident.

Streams#

Operation event and log streams over server-sent events feed both the dashboard and CLI follow commands. See Logs and status.

Verify#

The doctor report has no unresolved findings.
All six units are active.
midnight audit verify succeeds.
A mutating agent action appears in the audit record.

If it does not work#

Symptom	Check	Recovery
`audit verify` fails	Whether rows were altered or removed outside the application	Preserve the database before anything else; a failed verification is evidence
The doctor report lists findings	Each finding names its subject	Resolve before enabling units
Deploys queue but never build	Worker and buildkit units	Start them; check `journalctl`
A unit restarts repeatedly	`journalctl -u <unit>`	Fix the underlying cause rather than raising restart limits
Metrics are empty	Whether the workload is running	Metrics need a running service

Warning

If midnight audit verify fails, capture the database and logs before restarting anything. A broken hash chain is the strongest evidence you will get, and a restart or repair attempt can destroy it.