Runtime Host

Appearance

Shutdown semantics ​

This page covers two related things: how the Host shuts itself down, and the launch-readiness work that made shutdown and the subsequent restart safe — atomic writes, parse-and-skip bootstrap, idempotent seed, journald cap, and StartLimitBurst hardening. The work lives under WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/.

Host stop sequencing ​

matrix stop (CLI: cli.ts:140-143) calls stopRunningHost (cli.ts:960-998). For a live host.status.json:

  1. Try graceful supervisor shutdown. Send shutdown to host.supervisor over NATS with the configured timeout (default 10s). The supervisor schedules its own MatrixHostService.stop() on setImmediate and replies immediately (matrix-host-service.ts:838-844).
  2. Wait for pid exit up to timeoutMs.
  3. If the pid is still alive after the deadline: SIGTERM.
  4. Wait another 2s.
  5. If still alive: SIGKILL.

MatrixHostService.stop() (matrix-host-service.ts:106-123) does the graceful work in this order:

  1. Stop every managed runtime (parallel, stopRuntimeProcess per id).
  2. Stop the supervisor control server (close NATS subscription).
  3. Stop the embedded NATS sibling (only if mode: "embedded" and the Host owns the sibling — wrapper-managed siblings are not the Host's responsibility).
  4. Mark status.status = "stopping" atomically.
  5. Clear host.status.json and host.pid.

Each runtime's stop follows the Stop runtime sequence: graceful runtime.shutdown first, SIGTERM after 5s grace, SIGKILL after another 2s.

NATS sibling teardown (wrapper installs) ​

The wrapper supervises NATS as a fully detached sibling. hivecast stop (hivecast.mjs:1765-1770) calls stopHostProduct, which in turn calls stopNatsSibling (hivecast.mjs:687-723):

  1. Read pidfile; if no pid or pid not alive, declare "already stopped."
  2. SIGTERM, wait up to 3s. NATS flushes JetStream and unbinds ports.
  3. If still alive: SIGKILL, wait up to 1s.
  4. Remove the pidfile.

The 3-second SIGTERM grace is intentional: an immediate SIGKILL leaves JetStream mid-write and the next NATS start can race against still-bound ports — see the comment at hivecast.mjs:700-704.

Recovery work — what makes restart safe ​

1. Atomic JSON writes ​

projects/matrix-3/packages/host-service/src/util/atomic-write.ts:17-48 implements atomicWriteJson and atomicWriteText:

ts
function atomicWriteText(filePath: string, value: string): void {
  const dir = path.dirname(filePath);
  fs.mkdirSync(dir, { recursive: true });
  const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}`;
  let fd: number | null = null;
  try {
    fd = fs.openSync(tmpPath, 'wx', 0o644);
    fs.writeSync(fd, value);
    fs.fsyncSync(fd);
  } finally { /* close */ }
  fs.renameSync(tmpPath, filePath);
  // best-effort directory fsync for crash durability
}

Every Host JSON write — host.json, host.status.json, every runtime.json, every runtime-env/*.environment.json — goes through this path. A crash mid-write cannot leave a half-written file. The worst case is a leftover .tmp.<pid>.<ts> next to the real file, which is harmless.

2. Parse-and-skip bootstrap ​

HostStateStore.listRuntimeRecordsAndCorrupt (host-state-store.ts:106-142) replaces a "throw on first bad record" shape with a {records, corrupt} shape. Reasons:

  • A single zero-byte runtime.json (the original 22-hour crash-loop bug — see commit d270d625) used to crash bootstrap, which crashed the systemd unit, which kept restarting forever.
  • Now bootstrap succeeds with the healthy records and reports the corrupt list; hivecast seed re-seeds empty records on the next start; the Host stays up.

3. Idempotent seed ​

hivecast seed (hivecast.mjs:494-554) re-seeds the bundled package store from the wrapper's dist/node_modules/, scans existing runtime records, and emits a structured report:

json
{
  "ok": true,
  "home": "...",
  "seeded": true,
  "runtimeRecords": {
    "healthy": 9,
    "corrupt": [
      { "runtimeId": "...", "path": "...", "kind": "empty", "size": 0 }
    ]
  }
}

Running seed repeatedly is safe.

4. journald cap ​

The .deb postinst installs /etc/systemd/journald.conf.d/hivecast.conf with SystemMaxUse=2G, SystemKeepFree=4G, RateLimitIntervalSec=30s, RateLimitBurst=10000. See Logs. This is the disk-safety backstop for runaway log emission during a crash loop.

5. StartLimitBurst / StartLimitIntervalSec hardening ​

Both hivecast-nats.service and hivecast-host.service ship with:

ini
StartLimitBurst=5
StartLimitIntervalSec=60

(build-deb-installer.js:111-112 and 139-140.) systemd will refuse to restart the unit more than 5 times in 60 seconds, breaking the runaway-restart pattern that originally filled disks. After hitting the limit, an operator must systemctl reset-failed <unit> and investigate.

Orphan recovery on next start ​

Even with all of the above, a Host process that crashed while runtimes were live needs to reconcile. _recoverOrphanedAutoRuntimeRecordsBeforeStart (matrix-host-service.ts:578-622) runs first thing in start():

  • For every record with startup: "auto" and a live status:
    • If the recorded pid is alive but its commandline doesn't match, mark failed with a staleReason.
    • If the pid is alive and matches, SIGTERM, then SIGKILL after 2s. Mark the record stopped with orphanRecoveredAt.
    • If the pid refuses to die, refuse to start.
  • Then _autoStartPersistedRuntimes walks the records in priority order and respawns them fresh.

This is what makes systemctl restart hivecast-host.service safe even when the prior Host left some children behind.

See also ​