Matrix Package Lifecycle

Appearance

Health ​

A runtime's "health" combines four signals: the runtime's own lifecycle state, its heartbeat to system.runtimes, the Host's process supervision state, and the gateway's /healthz endpoint.

Runtime lifecycle states ​

Source: packages/mx-cli/src/utils/runner-runtime-control.ts.

starting → live → stopping → stopped
                    ↓
                  failed
StateMeaning
startingThe runtime is initializing; the factory has been called but mounts may not be ready
liveThe runtime is fully started and serving
stoppingA shutdown was requested; the runtime is winding down
stoppedThe runtime exited cleanly
failedThe runtime errored during startup or operation

The runtime publishes its state via the runner control plane to system.runtimes. hivecast runtimes reports the latest known state.

Reading lifecycle state ​

bash
hivecast runtimes --home /tmp/matrix-home
# or:
matrix invoke system.runtimes runtimes.list '{}'

Each entry includes runtimeId, state, packageRef, startedAt, and the runtime's mounts.

Heartbeat ​

Each --serve runtime starts a heartbeat timer:

ts
// packages/mx-cli/src/commands/run.ts:391-397
heartbeatTimer = setInterval(() => {
  void heartbeatRunnerRuntime(heartbeatRuntime, runtimeRegistration)
    .catch(() => undefined);
}, 10_000);

Every 10 seconds, the runtime calls heartbeatRunnerRuntime against the Host's runner control plane. system.runtimes records the last heartbeat. A runtime that misses heartbeats will show as stale even if its OS process is alive.

/healthz for the Host ​

The Host gateway exposes /healthz:

bash
curl -fsS http://127.0.0.1:3100/healthz | jq .

Returns the Host's view of itself: HTTP server alive, NATS sibling reachable, host-control mounted. CLAUDE.md "First 10 Minutes" documents this as the basic liveness check after install.

A /healthz 200 confirms the Host is up. It does not confirm any particular runtime is up; check hivecast runtimes for that.

hivecast status ​

bash
hivecast status --home /tmp/matrix-home

Reports the Host's stored status (from <host-home>/host.status.json): HTTP host/port, transport root, supervisor pid, NATS sibling pid. Does not enumerate runtimes; pair with hivecast runtimes.

hivecast doctor ​

For deeper diagnostics:

bash
hivecast doctor --home /tmp/matrix-home --json | jq .
hivecast doctor --home /tmp/matrix-home --repair

Audits the Host home for known failure modes (corrupt runtime.json, missing NATS binary, etc.). With --repair, attempts to self-heal. See projects/matrix-3/packages/docs-hivecast/content/cli/hivecast-host.md for the full check list.

Triage matrix ​

SymptomFirst checkThen
Browser tab won't loadcurl /healthzhivecast status, then hivecast runtimes for the right appName
hivecast runtimes shows failedTail the runtime's stderr logIf "MX_MANIFEST_INVALID", rebuild and re-up the package
Runtime is live but bus calls hangmatrix invoke against a known good actor (system.runtimes runtimes.list)If that hangs too, NATS is the issue; check journalctl -u hivecast-nats.service
Runtime is live but logs show nothingConfirm actorLog is being used; bare console.log lands in stdout but won't carry actor identityInspect system.observability if mounted
hivecast status reports stopped, but processes are aliveStale pidfile; hivecast stop followed by hivecast start

See also ​

Source: Lifecycle states defined in projects/matrix-3/packages/mx-cli/src/utils/runner-runtime-control.ts; heartbeat in packages/mx-cli/src/commands/run.ts:391-397.