Runtime Host

Appearance

Host lifecycle ​

A Host moves through a small, well-defined set of states. The state is recorded in <host-home>/host.status.json and the durable on-disk record is the source of truth — process liveness is checked against host.pid on read.

States ​

The IHostStatus.status field (projects/matrix-3/packages/host-service/src/types.ts:124) is one of:

  • "starting" — MatrixHostService.start() is running, NATS sibling has been spawned (or external NATS verified), but supervisor control has not yet bound. Brief; if you see it persist, look for a stuck NATS handshake.
  • "running" — supervisor control is bound on host.supervisor.PID-<pid>.$inbox, auto-start runtimes are being launched or are running. Steady state.
  • "stopping" — MatrixHostService.stop() has begun. Runtimes are being asked to shut down via their controlMount then SIGTERM'd if they don't acknowledge.
  • "stopped" — written briefly during stop() then host.status.json is removed. If you see this on disk it means the Host crashed mid-stop.

isLiveStatus in projects/matrix-3/packages/host-service/src/host-state-store.ts:1207-1209 treats "starting", "running", and "stopping" as live.

Install -> start -> run ​

text
1. hivecast install --home <home>
   - mkdir <home>; drop bundled nats binary; seed system package store
   - generate host.json, credentials/hivecast-install.json
   - register default runtime targets (manual or auto per --no-start)
   - if not --no-start: hivecast start

2. hivecast start --home <home>
   - acquire host.start.lock (cli.ts:1031-1057)
   - if status="running" already, refuse: { ok: true, alreadyRunning: true }
   - start NATS sibling (or verify external NATS reachable)
   - recover orphaned auto runtime records
     (matrix-host-service.ts:578-622)
   - write host.status.json with status="running"
   - bind supervisor control on host.supervisor.PID-<pid>.$inbox
   - auto-start runtimes whose record has startup="auto", in priority order
     (matrix-host-service.ts:494-541)

3. running
   - supervisor watches every runtime child; on exit,
     - record final status (stopped|failed) atomically
     - if restart policy applies, schedule a restart 1s later
       (matrix-host-service.ts:484-489)
   - host-control runtime publishes device.heartbeat every 15s

4. hivecast stop --home <home>
   - cli.ts:140-143 -> stopRunningHost
   - try host.supervisor "shutdown" RPC first (graceful)
   - on timeout, SIGTERM the host pid
   - on second timeout, SIGKILL
   - finally, stop NATS sibling

Crash recovery ​

Two recovery codepaths matter:

1. Orphaned auto-start runtimes ​

If the Host process crashes while live runtimes are still running, the next MatrixHostService.start() calls _recoverOrphanedAutoRuntimeRecordsBeforeStart (projects/matrix-3/packages/host-service/src/matrix-host-service.ts:578-622):

  • For every runtime record with startup: "auto" and a live status:
    • If the recorded pid is still alive but its commandline / cwd no longer matches the record, mark it failed with a staleReason.
    • Otherwise, SIGTERM the orphan, then SIGKILL after 2s. Mark the record stopped with orphanRecoveredAt metadata.
  • If any orphan refuses to die, refuse to start. The Host will not auto-spawn duplicates of a runtime whose old process is still consuming its mount.

2. Corrupt runtime records ​

Before this work, a single zero-byte runtime.json would crash start. Now HostStateStore.listRuntimeRecordsAndCorrupt (projects/matrix-3/packages/host-service/src/host-state-store.ts:106-142) parses-and-skips: returns the healthy records and the corrupt list separately. hivecast seed reports the corrupt list honestly. Auto-start re-seeds the record on the next start. See WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/ for the full work — atomic writes (every Host JSON file goes through atomicWriteJson), parse-and-skip recovery, journald cap, and unit hardening (StartLimitBurst=5, StartLimitIntervalSec=60).

State transitions diagram ​

mermaid
stateDiagram-v2
    [*] --> installed: hivecast install
    installed --> starting: hivecast start
    starting --> running: NATS up + supervisor bound
    starting --> failed: NATS or supervisor refused
    running --> stopping: hivecast stop / SIGTERM
    stopping --> stopped: graceful or signal-driven shutdown
    stopped --> [*]
    running --> running: runtime crash + restart policy
    failed --> starting: human re-runs hivecast start

failed is not a host.status.json value — it is what an operator sees from hivecast doctor when bootstrap returned an error.

See also ​