Troubleshoot PacketRelay#

Start with transport and instance identity, then read the WebSocket close code, then inspect hello admission. Most failures are caused by proxy upgrades, clients reaching different replicas, an invalid first frame, or a client that stopped reading its socket.

Before you start#

Obtain the relay hostname, client URL, session identifier, and approximate connection time.
Collect relay logs without copying application payloads into a ticket.
Install curl and a WebSocket client from a trusted workstation.

Check the service#

test -n "$PACKETRELAY_URL" || { echo "Set PACKETRELAY_URL to the deployed origin" >&2; exit 1; }
curl -i "$PACKETRELAY_URL/health"
curl -i "$PACKETRELAY_URL/ready"

/health returns 200 ok whenever the process is listening. /ready returns 200 ready normally and 503 draining from the moment the relay receives a stop signal until the process exits. A short burst of draining during a restart or deployment is expected; minutes of it means the shutdown is stuck.

If either check fails, fix the process, port, DNS, or TLS edge before testing WebSocket traffic.

Read the close code#

The relay states why it ended a connection. Each code carries a different correct client response, so branch on the code rather than reconnecting identically after every disconnect.

Code	Reason text	What happened	What the client should do
`1001`	`relay shutting down`	The relay is draining before exit.	Reconnect after a short backoff. Wait for `/ready` to return `ready`.
`1008`	`message budget exceeded`	The connection exhausted `--max-messages-per-window` or `--max-bytes-per-window`.	Reduce the send rate before reconnecting. An immediate retry loop is refused again.
`4000`	`replaced by a newer connection`	A newer connection took the same session or member slot.	Do not reconnect. Reconnecting fights the connection that now owns the slot.
`4001`	`heartbeat timeout`	No frame arrived within the heartbeat window.	Reconnect normally.
`4002`	`slow consumer`	The client stopped draining its outbound queue.	Reconnect, then consume faster or reduce the fan-out rate.

The relay never sends 1005, 1006, or 1015. A client that reports 1006 was disconnected by the transport or an intermediary, not by the relay — check proxy idle timeouts and the network path.

Diagnose heartbeat timeouts#

After --ping-interval seconds of inbound silence (25 by default) the relay sends a WebSocket ping and gives the peer --pong-timeout seconds (10 by default) to produce a frame. Any inbound frame counts, including the automatic pong most WebSocket libraries send, so a client only needs to keep reading its socket. With the defaults a vanished peer is closed with 4001 within 35 seconds — inside the idle timeout of a typical proxy edge.

A client that closes with 4001 while its user believes it is connected is usually blocked in its own event loop, suspended by the operating system, or behind a proxy that is buffering rather than forwarding frames.

Check the WebSocket path#

Open a socket and send a valid hello immediately. Route selection does not depend on the URL path, but the path still has to be one the relay leaves free. Requests answer 404 when the path is reserved — /ws/host, /ws/device, /api, /auth, /push, and anything under /api/, /auth/, or /push/ — and when the path merely begins with /health, /ready, or /metrics without matching a served path exactly. A request that is not GET over HTTP/1.1 is refused the same way. Connect clients to any remaining path, such as /; the full list is in Protocol.

Symptom	Check	Recovery
HTTP works, WebSocket does not upgrade	Whether the edge forwards WebSocket upgrade headers	Enable WebSocket proxying at the edge.
Upgrade answered `404 not found`	Whether the client path is reserved, or begins with a health or metrics prefix	Move the client to a free path such as `/`.
Upgrade refused with `503 draining`	Whether `/ready` reports `ready`	Retry once `/ready` returns `ready`.
Upgrade refused with `503 route capacity reached`	Current connections against `--max-connections`	Shed load or raise the reviewed limit.
Socket closes before any response	Whether the first text frame was a complete, supported JSON hello	Send a supported JSON hello as the first text frame.
`invalid session ID`	Identifier violates length or character rules	Use only letters, digits, `.`, `_`, and `-`.
`session not found`	A joining role arrived before the creating role	Connect the desktop or host first.
`session kind mismatch`	The identifier already belongs to another session kind	Generate a new identifier.
Peers both look ready but cannot see each other	Requests reached different relay replicas	Route both clients to the same instance.
Room admission fails	Signing payload, key format, signature format, or nonce is wrong	Re-run the room signing test vector and synchronise clocks.
Large traffic disconnects with `1008`	A per-connection traffic budget was exceeded	Reduce frame size or send rate, or adjust the reviewed limit.

Room frames rejected after the envelope#

Room fan-out wraps each payload in a member_frame envelope that names the relay-authenticated sender. The envelope adds a fixed 51 bytes plus the length of the sender's member_id, so a payload the relay accepted can still be too large to forward.

When that happens the sender stays connected and receives:

{"type":"error","message":"room frame exceeds message ceiling after envelope"}

The frame is not delivered to any member. Keep room payloads at or below --max-message-bytes minus 51 minus the byte length of your member_id. At the 65,536-byte default ceiling, a 16-byte member_id leaves 65,469 bytes. Characters that JSON escapes, such as " and \, occupy more bytes once encoded, so payloads containing them need more headroom than that arithmetic suggests.

Metrics scrape problems#

Symptom	Check	Recovery
`GET /metrics` returns `404 not found`	Whether a metrics token is configured at all — without one the route is disabled and looks like any unknown path	Set the token as a deployment secret and redeploy.
`GET /metrics` returns `404 not found` with a token configured	The exact scrape path — a query string or trailing slash makes it an unknown path	Scrape the exact path `/metrics`.
`GET /metrics` returns `401 unauthorized` with `WWW-Authenticate: Bearer`	Whether the request carried an `Authorization: Bearer` header with the matching token	Correct the scrape credential.

Verify#

After changing configuration or client behaviour:

Confirm /ready returns ready.
Create a fresh bridge identifier.
Connect both roles and exchange a frame in each direction.
Leave one connection idle for a minute and confirm it survives the heartbeat.
Disconnect and reconnect one role.
Confirm the remaining peer receives the expected lifecycle events.

Deploy — listener and proxy setup.
Protocol — accepted hellos and error envelopes.
Operations — limits, metrics, and replica behaviour.