Monitoring and metrics#

GET /api/admin/metrics exports Sugar's live operational state as Prometheus text exposition, so error rate, latency, breaker state, spend and quota pressure can page someone instead of waiting to be noticed in the portal. It is a projection of data Sugar already records; nothing in the gateway increments a counter for it.

Before you start#

An administrator account, or an sgr_admin_ automation token carrying the usage:read scope.
A scraper that can send a bearer token, or curl and jq.

The endpoint is never anonymous. A scrape names every model in use, every member's budget position and which lanes are broken, so treat it as an operational disclosure even though it carries no prompt or completion text.

The endpoint#

Property	Value
Path	`GET /api/admin/metrics`
Default format	Prometheus text exposition, version 0.0.4
JSON	`GET /api/admin/metrics?format=json`
Window	`?window=<seconds>`, clamped to 60–86400, default 300
Caching	`cache-control: no-store`
Authentication	Admin browser session, or an `sgr_admin_` token with `usage:read`

Create a scrape token#

Open Admin → Tokens, create a token with the usage:read scope and the shortest expiry your scrape schedule tolerates. Tokens expire within 90 days. Write the value to a file only the scraper can read; it is shown once.

Configure a scraper#

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: sugar
    metrics_path: /api/admin/metrics
    scheme: https
    static_configs:
      - targets: ["sugar.example.invalid"]
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/sugar-metrics.token

Keep the scrape interval at or below the metrics window so no traffic falls between two scrapes. Scraping faster than about 15 seconds re-reads the same rows without resolving anything finer.

Without a scraper, poll the JSON form:

curl -sf -H "Authorization: Bearer $SUGAR_METRICS_TOKEN" \
  "$SUGAR_URL/api/admin/metrics?format=json"

Read the numbers correctly#

Important: the windowed series are gauges, not counters.

Series ending _recent cover the rolling window reported by sugar_metrics_window_seconds. Never wrap one in rate() or increase(); divide by the window for a per-second rate.
The latency histograms are gauge histograms. Apply histogram_quantile() directly to the bucket series, never through rate().
sugar_workspace_attributed_cost_usd is what the month's work was worth at metered rates and is the figure every cost ceiling counts, including work a prepaid lane served. sugar_workspace_spend_usd is money actually paid and is zero for prepaid lanes. A dashboard showing only spend reports a workspace at zero while it is being refused for hitting its ceiling.

Series worth alerting on#

Series	Meaning
`sugar_gateway_requests_recent{outcome}`	Requests in the window by outcome; all seven outcomes are always exported, at zero when unseen.
`sugar_gateway_requests_by_status_recent{status_class}`	The same rows by HTTP status class.
`sugar_gateway_request_duration_seconds`	End-to-end latency histogram.
`sugar_gateway_time_to_first_token_seconds`	Time to the first visible token, observed only for requests that produced one.
`sugar_breaker_blocked_lanes`	Count of lanes currently withholding traffic — tripped and not yet due for a probe. Zero means every lane is serving.
`sugar_breaker_state{provider,provider_model}`	Per-lane breaker state: `0` closed, `1` half-open, `2` open. Only a lane that has failed at least once has a row, so an absent lane is a healthy lane.
`sugar_quota_utilization_ratio{scope,scope_id}`	Fraction of the tightest budget a scope has consumed.
`sugar_route_surface_missing_routes`	Critical routes this build did not register; non-zero means `/api/health` is already 503.
`sugar_metrics_scrape_duration_seconds`	Time this scrape spent reading SQLite.

sugar_breaker_blocked_lanes carries no labels; it is one number. The per-lane breaker series are the labelled ones, so a selector such as sugar_breaker_blocked_lanes{provider="..."} matches nothing. Alert on the count, then use sugar_breaker_state to name the lanes.

Per-lane breaker series are capped at 100, with lanes that are not closed kept first, so the cap can only drop healthy rows. Per-member quota series are capped, default 25, by SUGAR_METRICS_MEMBER_SERIES (0–200). The workspace roll-up is always exported. Lower the cap or the window first if scrape duration climbs.

Verify#

Scrape once with the token and confirm HTTP 200 and a text/plain body whose first lines carry sugar_build_info. Confirm sugar_metrics_window_seconds matches the window you asked for. Repeat with no credential and confirm 401, then with a member session and confirm 403.

If it does not work#

401. No credential, or a bearer that is not an admin automation token.
403 admin_token_scope_forbidden. The token lacks usage:read.
403. A signed-in member session; the endpoint requires an administrator.
Alerts read no data. Check the query for rate() on a _recent series or a histogram bucket.