> ## Documentation Index
> Fetch the complete documentation index at: https://reversi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Health, Prometheus metrics, structured logs, alerts and OpenTelemetry traces.

## Health

| Endpoint | Meaning |
| - | - |
| `GET /healthz` | The process is up |
| `GET /readyz` (server) | The store answers and the ledger head can be read; 503 otherwise. Also reports whether the kill switch is on |
| `GET /readyz` (gateway) | Every upstream answers a ping, with per-upstream status |

## Metrics

`GET /metrics` on the server, in Prometheus text format. It needs an `auditor` credential (a bearer token in your scrape
config), unless `REVERSI_METRICS_PUBLIC=1` (only on a private network).

| Metric | Type | Meaning |
| - | - | - |
| `reversi_http_requests_total{method,route,status}` | counter | Requests, by matched route (ids never become labels) |
| `reversi_http_request_duration_seconds` | summary | Handling time |
| `reversi_ledger_events_total{event}` | counter | Ledger records by event: `intended` is calls, `blocked`, `failed`, `compensated`, `compensation_failed`... |
| `reversi_ledger_records` | gauge | Ledger length |
| `reversi_runs` | gauge | Runs in the store |
| `reversi_approvals{status}` | gauge | Approvals by status |
| `reversi_approval_latency_seconds` | summary | Request to decision |
| `reversi_halted` | gauge | 1 while the kill switch is on |
| `reversi_uptime_seconds` | gauge | |

Useful alerts: `reversi_halted == 1`, a rising rate of `reversi_ledger_events_total{event="compensation_failed"}`,
`reversi_approvals{status="pending"}` above zero for longer than your approval SLA, and `readyz` failing. Most of these are
also available as built-in alerts below, delivered to the places you already watch.

## Logs

`REVERSI_LOG=json` makes the server write one line per request:

```json theme={null}
{"ts":"2026-10-16T17:21:10.820Z","method":"POST","path":"/v1/approvals/apr_…/approve","status":200,"ms":21.1,"principal":"fred"}
```

Bodies, headers and credentials are never logged. The ledger itself is the audit log.

For your own code, `createLogger` from `@multimindlab/reversi` writes one JSON object per line
(`{"level","time","msg",…fields}`) at `debug`, `info`, `warn` or `error`, with `child` loggers for fixed fields. It redacts
before writing: credential patterns inside any string, and the whole value of any field whose name looks like a secret
(`password`, `secret`, `token`, `authorization`, `api_key`, `cookie`, `credential`, `private_key`, `signature`). The `log:`
section of `reversi.config.yaml` configures the engine's own logger, which writes reloads, failed notifications, anchors and
trace export errors to stderr:

```yaml reversi.config.yaml theme={null}
log: { level: info, format: json }     # level: debug | info | warn | error; format: json | pretty
```

`json` writes redacted JSON lines (`{"level":"warn","time":"…","msg":"…"}`); `pretty` writes `reversi [warn] message`.
`REVERSI_LOG=json` is separate: it only controls the per-request line of the server.

## Alerts

Alerts are conditions worth a human, computed from the ledger, run state, approvals and (on the gateway) upstream health.
They are always visible: the console **Alerts** page and `GET /v1/alerts` (auditor and up) evaluate them live. When
[notification channels](/operations/notifications) are configured, an alert manager also sends them.

| Kind | Severity | Fires when |
| - | - | - |
| `compensation_failed` | critical | A step's compensation failed and the run is not rolled back |
| `verification_failed` | critical | An undo ran but reading back its effect failed |
| `break_glass` | critical | A [break-glass](/concepts/approvals) approval was recorded |
| `upstream_unhealthy` | critical | A gateway upstream does not answer its ping |
| `budget_threshold` | warning, critical at 100% | A budget crossed a threshold |
| `rollback_unverified` | warning | An undo ran but could not be verified |
| `irreversible_executed` | warning | An irreversible tool succeeded |
| `unclassified_mutation` | warning | A tool with no manifest was called |
| `policy_denial_spike` | warning | Blocked calls reach the threshold (default 10) in the window |
| `unresolved_outcome` | warning | A step has been `intended` or `uncertain` for over an hour |
| `approval_timeout` | warning | An approval expired unanswered or escalated |
| `halted` | warning | A scoped halt is active |

Event-based kinds look at the last **15 minutes** of the ledger, so they clear on their own. `compensation_failed`,
`unresolved_outcome` and `halted` are state-based and stay until fixed or lifted.

```yaml reversi.config.yaml theme={null}
alerts:
  interval: 1m          # how often to evaluate (default 1m)
  remind: 1h            # repeat a still-active alert (default 1h)
  denial_spike: 10      # blocked calls in the window (default 10)
```

The manager sends an alert **once** when it appears, again every `remind` while it persists, and a `Resolved:` message
(which closes the PagerDuty or Opsgenie incident) when the condition clears. State is kept in the data store, so several
processes sharing it do not repeat each other. `denial_spike` is honoured by the manager, `GET /v1/alerts` and the console Alerts page alike. The window and the one-hour unresolved limit are not configurable in the
project file. The manager only runs when at least one channel is configured; with none, use the console or `GET /v1/alerts`.

## Tracing

Reversi exports OpenTelemetry traces over OTLP/HTTP with JSON. There is no SDK to install.

```yaml reversi.config.yaml theme={null}
otel:
  endpoint: http://collector:4318       # spans go to <endpoint>/v1/traces
  service_name: reversi
  headers: { x-api-key: "${OTEL_KEY}" }
```

`REVERSI_OTLP_ENDPOINT` sets the endpoint without a file. The server and gateway export while running.

| Span | From | To | Status |
| - | - | - | - |
| `reversi.call <tool>` | `intended` | `succeeded`, `failed` or `uncertain` | OK only if `succeeded` |
| `reversi.compensate <tool>` | `compensating` | `compensated` or `compensation_failed` | OK only if `compensated` |
| `reversi.call.blocked <tool>` | the block | the block | error for `blocked`, OK for `would_block` |

All spans of a run share one trace id (derived from the run id), so a run is one trace. Attributes:
`reversi.run_id`, `reversi.step`, `reversi.tool`, `reversi.actor`, `reversi.effect` (from the manifest),
`reversi.event`, `reversi.reason`, `reversi.detail`, `reversi.seq.start` and `reversi.seq.end` (the ledger positions, so a
span links back to the evidence). The resource carries `service.name` (default `reversi`) and `deployment.environment`
(from `environment:`). Spans carry names and ids, never arguments or results.

Spans are sent in batches of 50 or every five seconds, with a 10 s timeout, through the
[egress guard](/operations/retention). A failing collector is logged to stderr and never affects calls.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.