Runtime Host

Appearance

Runtime status ​

A runtime record's status field (projects/matrix-3/packages/host-service/src/types.ts:85) is one of five values:

StatusMeaningLive?
startingspawn returned, readiness probe in progressyes
runningreadiness probe passed; supervisor expects this runtime aliveyes
stoppingmatrix down in flight, graceful shutdown sentyes
stoppedruntime exited cleanly (exit code 0 or SIGTERM)no
failedruntime crashed or readiness timed outno

isLiveRuntimeStatus (matrix-host-service.ts:1211-1213) returns true for starting, running, stopping. The launch-readiness work's liveness checks (see below) use this to decide whether a record is even claiming to be alive.

Transitions ​

mermaid
stateDiagram-v2
    [*] --> starting: matrix up
    starting --> running: runtime.ready ok
    starting --> failed: readiness timeout / spawn error
    running --> stopping: matrix down
    running --> stopped: child exit(0) without down
    running --> failed: child exit(N) without down / SIGKILL
    stopping --> stopped: graceful shutdown ok / SIGTERM clean exit
    stopping --> failed: SIGKILL after grace period (rare)
    failed --> starting: matrix up (operator action) / auto-start sweep
    stopped --> starting: matrix up / auto-start sweep

The Host writes the new record atomically on every transition. updatedAt ticks (ISO8601 string) so callers can detect change. startedAt is written once on the first successful spawn. stoppedAt is written when the status leaves a live state.

Where status can drift ​

A runtime.json says status: "running" but the real process is dead. Three reasons this happens:

  1. The Host crashed before it could update the record.
  2. The pid was recycled by another process after the Host died.
  3. The pid is still alive but for a different commandline (someone else restarted the package out-of-band).

The Host never trusts a record blindly. Three reconciliation passes:

1. _refreshRuntimeRecordLiveness ​

Called whenever listRuntimeRecords is invoked (matrix-host-service.ts:234-240). For any record claiming a live status:

  • If pid is missing -> mark failed with reason "live runtime record has no pid".
  • If pid is no longer alive -> mark failed, "recorded runtime pid is no longer alive".
  • If pid is alive but runtimeProcessIdentityMismatchReason (matrix-host-service.ts:1107-1131) returns a non-empty reason (commandline / args / cwd diverged) -> mark failed, "recorded runtime pid belongs to a different ...".

2. _refreshRuntimeRecordControlLiveness ​

Called from listRuntimeRecordsWithControlLiveness (matrix-host-service.ts:269-309). For records still claiming live state, invoke runtime.health over the controlMount with a 1.5s deadline. If the runtime does not respond, or replies with { healthy: false }, mark the record controlHealthStatus: "degraded" plus a controlHealthReason. The status itself stays running; degraded just flags it.

3. _recoverOrphanedAutoRuntimeRecordsBeforeStart ​

Runs at the very start of MatrixHostService.start() (matrix-host-service.ts:578-622). For any auto-start record claiming live status with a live but unmanaged pid: SIGTERM, then SIGKILL after 2s, then mark stopped with orphanRecoveredAt. If a pid refuses to die, the Host refuses to start and reports the failure list. This prevents accidentally double-running an auto-start runtime when the Host process restarts and the previous runtime survived.

Reading status ​

bash
# Quick view, all runtimes.
matrix runtimes | jq '.runtimes[] | { id: .runtimeId, status: .status, pid: .pid }'

# Filter to failed only.
matrix runtimes | jq '.runtimes[] | select(.status=="failed")'

# Including degraded (control-mount-unresponsive) records, ask via host-control.
matrix invoke host.control runtime.list '{}' | jq '.runtimes[] | { id: .runtimeId, status, controlHealthStatus: .metadata.controlHealthStatus }'

The CLI's status and runtimes commands route through invokeHostControlFromStatus when the Host is live (cli.ts:53-98), and fall back to reading the on-disk records directly if the supervisor is unreachable.

See also ​