Troubleshooting#

Symptom-led recovery, fastest check first.

Start here#

Three checks, in order:

cat /var/lib/midnight/doctor-postinst.json

systemctl status midnight-server midnight-worker midnight-buildkit

midnight whoami

The doctor report tells you whether the host is sound; unit status tells you whether the platform is running; whoami tells you which identity your commands are using. Most reports resolve to one of those three.

Installation#

Symptom	Check	Recovery
Docker errors during install or build	Whether the distribution Docker package is installed	It is not supported; use upstream `docker-ce`
Docker API rejected	API version	v1.43 or newer is required
Ports 80 or 443 unavailable	What holds them	Free them, or point Midnight at your existing Traefik
Migrations did not run	Whether a connection string was configured at install	Configure it and re-run available migrations
Units fail immediately	The doctor report	Resolve findings before enabling units
Cgroup or access-control errors	Cgroup v2, and AppArmor or SELinux state	Both are prerequisites, not optional

Authentication and scope#

Symptom	Check	Recovery
`login` fails	The API base URL	It must be your installation's API base
`up` refuses with a credential error	Session scope	`up` never falls back to a recovery credential
A command is rejected before any request	Whether a non-human token was supplied	Use a human login, or the stored recovery credential for legacy routes
`link` finds no projects	Whether your identity can reach them	Discovery is tenant-scoped
An environment update is refused	Write permission on the exact project, and the expected version	Both are required

Build and deploy#

Symptom	Check	Recovery
Deploy queues and never builds	Worker and buildkit units	Start them
Build fails on autodetection	Whether a Dockerfile is present	Add one rather than fighting detection
GitHub source checkout fails	The repository grant and the link version	Grants are workspace-exclusive and repository-exact; the link version is rechecked at safe points
A retry created a duplicate	Whether the idempotency key was reused	Retry with the same key and unchanged arguments
Rollback did not fix it	Whether a migration or variable also needs reversing	Rollback covers the deployment only
An action reported success but changed nothing	Whether the endpoint returns a planning artefact	Some demo and plan endpoints do not act

Routing and TLS#

Symptom	Check	Recovery
Service unreachable	Whether durable TLS evidence exists	Routes fail closed by design
Traefik ignores the route	Traefik's static file-provider path	It must point at the Midnight dynamic directory
Issuance never completes	The DNS-01 provider configuration	Manual DNS-01 is available
Production issuance refused	The staging-first acknowledgement gate	Acknowledge deliberately after staging succeeds
Cloudflare records unchanged	The record connector is not implemented	Manage records yourself

Variables#

Symptom	Check	Recovery
The application sees an old value	Whether it has deployed or restarted since	Saving desired state does not deploy
A write is refused	The current version from `variable list`	Compare-and-swap requires the current version
`set` fails on an existing name	`set` is create-only for humans	Use `update`
A sealed value cannot be read	Sealed is write-only	Replace it rather than reading it
Secrets undecryptable after a restore	Whether the host key material matches the database	Restore the matching key set — see Backups and upgrades

Agent and MCP#

Symptom	Check	Recovery
A mutating tool is refused	The mutation gate	Refusal with mutations disabled is correct behaviour
No diagnosis on a failure	Whether it was a deploy or build outcome, and the time budget	Diagnosis is time-bounded
MCP requests throttled	Per-session rate limiting	Reduce concurrency
MCP rejected outright	The bearer token	Rotate and re-supply

Preserve before you retry#

Before restarting a unit, restoring a backup, or redeploying over a failure:

capture journalctl for the affected units;
capture the deployment record and its state; and
if midnight audit verify failed, capture the database first. A broken hash chain is evidence, and a restart or repair attempt can destroy it.