Runtime Host

Appearance

Runtime health ​

A runtime is "healthy" when it responds correctly to runtime.health on its controlMount with a matching runtimeId and pid. That is the only check the Host treats as authoritative.

The contract ​

Every runtime declares a controlMount (defaulting to system.runtimes.<runtimeId>.control). The runtime is expected to implement at least these three ops on that mount:

OpPurpose
runtime.ready"Are you done initializing? Return { ready: true }." Used at startup readiness gate.
runtime.health"Are you healthy right now? Return { healthy: true | false, state? }." Used at observation time.
runtime.shutdown"Please exit cleanly." Used at stop.

The Host invokes these via invokeRuntimeControl (projects/matrix-3/packages/host-control/src/matrix-actor-invoke.ts). Every response must include runtimeId and pid matching the supervised process; mismatches are flagged via invalidRuntimeControlResponseReason (matrix-host-service.ts:1062-1085).

Health probe at observation time ​

MatrixHostService._refreshRuntimeRecordControlLiveness (matrix-host-service.ts:269-309) runs whenever listRuntimeRecordsWithControlLiveness is called — which the supervisor's status and runtime.list ops do (matrix-host-service.ts:802-807). For each runtime in a live state:

  1. If pid is missing, no controlMount, or pid is not alive: skip.
  2. Otherwise, invoke runtime.health with a 1.5s deadline.
  3. If the response is missing/invalid, mark the record degraded with a controlHealthReason.
  4. If the response says healthy: false, mark the record degraded with runtime.health returned unhealthy state=....
  5. If the response is fine, leave the record alone.

Degraded marking does not change status — a degraded runtime is still running. It adds three metadata fields:

json
"metadata": {
  "controlHealthCheckedAt": "2026-05-04T18:30:01.512Z",
  "controlHealthStatus": "degraded",
  "controlHealthReason": "runtime.health did not respond: timeout"
}

These fields are stable across reads — every refresh that gets a clean response can clear them by writing a fresh record without those keys (though the current code only writes them on degrade — a "healthy" recheck does not erase past degrade markers; rely on controlHealthCheckedAt timestamp for currency).

Why HTTP 200 is not proof ​

A runtime serving a webapp can answer GET / with 200 OK while its NATS handshake is broken. The browser sees the HTML, then fails its WebSocket connect to /nats-ws, and the user sees a blank page with console errors. This is the failure mode called out repeatedly in the repo AGENTS.md § "Worker Bootstrap Gotchas" — "HTTP 200 is not proof."

runtime.health covers what HTTP 200 cannot:

  • The runtime's MatrixRuntime is connected to NATS.
  • Its actors are mounted.
  • Its runtimeId and pid match the supervised process.

If you rely only on curl /healthz, you can miss a runtime whose actor mount silently never came up.

Readiness probe at start time ​

_waitForRuntimeControlReady (matrix-host-service.ts:691-737) is the parallel codepath used during startRuntimeProcess. It polls runtime.ready every 100ms until the runtime returns a healthy response or the deadline elapses. Default deadline is 30s (matrix-host-service.ts:442), overridable via --timeout or spec.readyTimeoutMs.

If readiness times out, the Host:

  • Stops the child.
  • Reads up to 4 KiB of stderr + 2 KiB of stdout from the runtime's log files (appendRuntimeLogTail, matrix-host-service.ts:1322-1334).
  • Persists the record as failed with failedReason containing both the timeout message and the log tail.

This means a startup error is visible without leaving the supervisor prompt; you do not have to go reading log files separately.

Querying health from outside ​

bash
# Per-runtime health, via control mount directly.
matrix invoke system.runtimes.RUNTIME-HOST-LOCAL-ABC-CHAT.control runtime.health '{}'

# Aggregate via host-control.
matrix invoke host.control runtime.list '{}' | jq \
  '.runtimes[] | { runtimeId, status, ch: .metadata.controlHealthStatus, why: .metadata.controlHealthReason }'

# Aggregate via host-supervisor (skips host-control).
hivecast status --control supervisor 2>/dev/null \
  || matrix status --control supervisor

The --control supervisor flag (cli.ts:338-344) bypasses the host-control actor and talks straight to host.supervisor over NATS. Useful when host-control is itself the runtime in trouble.

What runtimes should do ​

Package authors implementing runtime.health should make it cheap and honest:

  • Return { healthy: true, state: "running", runtimeId, pid } quickly.
  • Return { healthy: false, state: "degraded", reason: "..." } if any invariant is broken (NATS disconnected, JetStream stream missing, required service unreachable).
  • Do not block on user-facing work or external network calls.
  • Always include runtimeId and pid; the Host's validator will mark the record degraded if either is missing.

See also ​