Runtime Host

Appearance

Failure modes ​

A small catalog of known failure shapes, the signatures they leave, the fix, and what prevents recurrence.

1. Zero-byte runtime.json (the original 22-hour crash loop) ​

Signature. A file at <home>/runtimes/<id>/runtime.json exists with size 0. Bootstrap parses it as JSON, fails, and (in older code) crashed start(). systemd's Restart=on-failure looped forever, journald filled the disk.

Repro path. Power loss or SIGKILL in the middle of a non-atomic fs.writeFileSync. (No longer reachable in current code; documented for archaeology.)

Fix.

  • hivecast seed reports the corrupt list explicitly (hivecast.mjs:494-554).
  • HostStateStore.listRuntimeRecordsAndCorrupt (host-state-store.ts:106-142) skips the bad record without throwing, so bootstrap completes.
  • The Host re-seeds the record from desired state on the next auto-start sweep if the runtime is startup: "auto".

Prevention.

  • All Host JSON writes go through atomicWriteJson / atomicWriteText (util/atomic-write.ts:17-48): wx open + fsync
    • rename + dir-fsync. A crash in the middle leaves the previous file intact plus a harmless .tmp.<pid>.<ts>.
  • systemd StartLimitBurst=5 / StartLimitIntervalSec=60 caps the restart rate.
  • journald drop-in caps log volume at 2 GiB.

The full brief is WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/.

2. Stale host.status.json after a crash ​

Signature. host.status.json says status: "running" with a pid that's no longer alive. hivecast start would otherwise refuse with alreadyRunning: true.

Fix. MatrixHostService.readStoredStatus (matrix-host-service.ts:125-135) checks isProcessAlive(status.pid) on every read. A dead pid causes clearStatus, which removes both host.status.json and host.pid.

Prevention. The check happens automatically; just running hivecast start clears the stale file. If something is so broken that even that fails, manually delete host.status.json and host.pid.

3. Orphaned auto-start runtime after Host crash ​

Signature. runtime.json says status: "running" with a live pid that doesn't belong to this Host process. After hivecast start, two copies of the same runtime would be running.

Fix. _recoverOrphanedAutoRuntimeRecordsBeforeStart (matrix-host-service.ts:578-622) runs first thing in start():

  • Compares the recorded pid's commandline / cwd against metadata.command/args/cwd (runtimeProcessIdentityMismatchReason, matrix-host-service.ts:1107-1131).
  • If identity matches, SIGTERM, then SIGKILL after 2s. Mark the record stopped with orphanRecoveredAt.
  • If identity does not match (the pid was recycled), mark the record failed with a staleReason and leave the unrelated process alone.
  • If the orphan refuses to die, refuse to start. The Host will not spawn duplicates.

Prevention. The check is unconditional; you cannot opt out.

4. Mount conflict at start ​

Signature.

Runtime CHAT-2 cannot start: logical mount "chat" is already owned by RUNTIME-HOST-LOCAL-ABC-CHAT

Fix. _findRuntimeStartConflict (matrix-host-service.ts:543-576) refuses to start a runtime whose localMounts (excluding system.runtimes.<id> private mounts) or webapp.routePrefix collides with another live runtime.

Resolution. matrix down the conflicting runtime first, or pick a different mount via --mount.

Prevention. Singleton system mounts (system.runtimes, system.registry, system.devices, system.auth, host.control) must have one live owner per authority root. The Host enforces this mechanically.

5. Readiness probe timeout ​

Signature.

Timed out waiting for runtime control actor system.runtimes.RUNTIME-X.control for RUNTIME-X
stderr:
[runtime] Error: connect ECONNREFUSED nats://127.0.0.1:4270

The Host attached the runtime's stderr tail (appendRuntimeLogTail, matrix-host-service.ts:1322-1354) so the real cause appears in the failure message.

Fix. Read the appended stderr. Common roots:

  • NATS not running. Check journalctl -u hivecast-nats.service or <home>/logs/nats.stderr.log.
  • Runtime initialization bug. Check the runtime's stderr at <home>/logs/runtimes/<id>/stderr.log.
  • Wrong NATS URL in the runtime's environment file. Look for stale values; trigger a config.reload to regenerate.

Prevention. The Host's pre-start NATS health check (waitForPorts, matrix-host-service.ts:1459-1482) fails loudly before any runtimes spawn if NATS itself didn't bind.

6. Identity mismatch at liveness refresh ​

Signature. A record drops to status: "failed" with a metadata.staleReason such as "recorded runtime pid belongs to a different command" (or "... different command line", or "... different working directory").

Fix. Someone else (operator, container restart, kernel pid recycling) replaced the process under the recorded pid. The Host refuses to claim a process it didn't spawn. Mark failed, do not signal the unrelated pid.

Resolution. Identify the replacement process; if it's still the runtime you want, stop and re-matrix up it cleanly so the Host re-records the right pid.

7. NATS sibling refused to start ​

Signature.

nats-server exited early (1): listen tcp 127.0.0.1:4270: bind: address already in use

Fix. Another nats-server (or any process) has the port. Find it (sudo ss -tlnp | grep 4270), stop it, retry.

Prevention. ensurePortsAvailable (matrix-host-service.ts:1484-1490) checks both NATS ports before spawning. If the wrapper's NATS sibling is already running, the next startNatsSibling call returns alreadyRunning: true instead of refusing.

8. Lock file held by dead process ​

Signature.

Matrix Host Service startup already in progress for /var/lib/hivecast (pid 12345)

Fix. acquireHostStartLock (projects/matrix-3/packages/host-service/src/cli.ts:1031-1057) detects this: if host.start.lock exists and its pid is dead, it removes the lock and retries.

If you're hitting this with a live pid, two hivecast start processes are racing; that's the lock doing its job.

9. Disk full from runaway logs ​

Signature. journald says No space left on device. Host can't write host.status.json. NATS can't fsync.

Fix.

  • Stop the unit causing the spam.
  • Truncate the journal: sudo journalctl --vacuum-size=500M.
  • Truncate per-runtime logs: sudo truncate -s 0 /var/lib/hivecast/logs/runtimes/<id>/stderr.log.

Prevention. journald drop-in (/etc/systemd/journald.conf.d/hivecast.conf) caps SystemMaxUse=2G. Per-unit LogRateLimitIntervalSec=10s / LogRateLimitBurst=200. Per-runtime files are not rotated; for production deployments add logrotate for <home>/logs/runtimes/*/.

10. Wrapper / .deb version skew ​

Signature. The .deb was upgraded but apt remove left /var/lib/hivecast/host.json pointing at a NATS port that the new postinst clobbered. NATS binds the new port, the Host can't connect because its config still says the old port.

Fix. Re-run the postinst's host.json writer by re-installing the package, or hand-edit host.json to match /var/lib/hivecast/nats/service/nats-server.conf.

Prevention. The .deb postinst overwrites both files deterministically. The wrapper does not. If you mix them on one home, expect drift.

See also ​