Runtime records

A runtime record is a JSON file that captures one runtime's identity, state, and policy. The Host treats these files as the authoritative source for "what runtimes exist" — the system.runtimes actor's in-memory view is a projection, not the truth.

This page documents the schema, lifecycle, and the corruption-recovery story (which has a real history — the original 22-hour zero-byte-runtime.json crash loop bug lives in this surface).

On-disk location

<host-home>/runtimes/<runtime-id>/runtime.json

One directory per runtime, one runtime.json per directory. The directory may also contain auxiliary state — but runtime.json is the canonical file.

<runtime-id> is sanitized to [A-Za-z0-9_-]+. Anything else is rejected at write time (sanitizeRuntimeId in projects/matrix-3/packages/host-service/src/host-state-store.ts:165-171).

Schema

From projects/matrix-3/packages/host-service/src/types.ts:79-96:

interface IHostRuntimeRecord {
  readonly runtimeId: string;
  readonly packageRef?: string;
  readonly packageDir?: string;
  readonly environment?: string;
  readonly pid?: number;
  readonly status: 'starting' | 'running' | 'stopping' | 'stopped' | 'failed';
  readonly startup?: 'manual' | 'auto';
  readonly restart?: 'never' | 'always' | 'on-failure';
  readonly startedAt?: string;
  readonly stoppedAt?: string;
  readonly updatedAt: string;
  readonly runtimeMount?: string;
  readonly controlMount?: string;
  readonly runtimeWireRoot?: string;
  readonly localMounts: readonly string[];
  readonly metadata: Record<string, unknown>;
}

Field reference

Field	Required	Meaning
runtimeId	yes	Unique id, e.g. RUNTIME-HOST-LOCAL-1F746E58793D-DIRECTOR
packageRef	one-of	Resolvable package name (e.g. @open-matrix/director); set if the runtime came up via package store
packageDir	one-of	Absolute path to a package directory; set if the runtime came up against a folder
environment	optional	Environment name (e.g. dev, hivecast); maps to <host-home>/runtime-env/<env>/<runtime-id>.env
pid	when running	OS pid of the running process
status	yes	One of starting, running, stopping, stopped, failed
startup	optional	auto (restore on Host start) or manual
restart	optional	Crash policy: always, on-failure, never
startedAt	when running	ISO-8601 timestamp
stoppedAt	when stopped	ISO-8601 timestamp
updatedAt	yes	Last update — written on every state transition
runtimeMount	optional	The runtime's system.runtimes.<id> management mount
controlMount	optional	The runtime's control plane mount, e.g. system.runtimes.<id>.control
runtimeWireRoot	optional	Per-runtime override of the Host's wire root; required for cross-authority setups
localMounts	yes	Array of logical mounts this runtime publishes (e.g. ["chat", "chat.conversation"])
metadata	yes	Free-form metadata. Common keys: command, args, cwd, webapp.routePrefix, webapp.servePort, staleDetectedAt, staleReason

localMounts and metadata are required (empty array / empty object permitted). localMounts is what system.registry and the gateway introspect to route requests; metadata.webapp.routePrefix is what the gateway uses to mount a webapp at /<routePrefix>/.

Status transitions

            up
   (none) ─────► starting
                    │
              healthy
                    ▼
              running
                    │
                  ┌─┴─┐
        crash  ──┤   ├──  down
                  │   │
                  ▼   ▼
              failed  stopping
                          │
                       (process exits)
                          ▼
                       stopped

• starting is short-lived — the supervisor sets it during process spawn. If the spawn succeeds and the runtime registers with system.runtimes, it transitions to running.
• running is the steady state.
• stopping is set when hivecast down is in progress; the supervisor signals the process and waits.
• stopped is the final state after a clean stop.
• failed is the final state after a crash whose restart policy doesn't allow recovery, OR when runtimeRecordStaleReason detects the recorded pid is no longer alive / belongs to a different process.

Reading records

hivecast runtimes

hivecast runtimes --home /tmp/matrix-home

Lists every record with summary fields. Delegates to host-service runtimes.

Direct file inspection

ls /tmp/matrix-home/runtimes/
cat /tmp/matrix-home/runtimes/RUNTIME-HOST-LOCAL-1F746E58793D-DIRECTOR/runtime.json | jq .

Through the actor surface

hivecast invoke system.runtimes runtimes.list '{}' --home /tmp/matrix-home
hivecast invoke system.runtimes runtimes.get '{"runtimeId":"RUNTIME-HOST-LOCAL-XXXX-DIRECTOR"}' --home /tmp/matrix-home

Both give the same data — the actor reads from the same on-disk records. Use the actor surface from running code; use the file surface for debugging.

Atomic writes

Records are written via atomicWriteJson (projects/matrix-3/packages/host-service/src/util/atomic-write.ts). The pattern:

1. Write to a temporary file <runtime.json>.<pid>.<timestamp>.tmp
2. fs.renameSync (atomic on POSIX) onto the destination

This prevents the original 22-hour zero-byte-runtime.json crash loop. The earlier code wrote directly with fs.writeFileSync, which on a crash mid-write left a zero-byte file; bootstrap then refused to start.

See WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/ for the full incident analysis.

Corruption recovery

HostStateStore.listRuntimeRecordsAndCorrupt (host-state-store.ts:106-142) is the parse-and-skip variant that bootstrap uses. It returns { records, corrupt }:

corrupt[].reason	Meaning
missing	The directory exists but runtime.json doesn't (silently skipped, partial mkdir before write)
empty	The file is zero bytes
parse	The file isn't valid JSON
shape	The file is JSON but missing required fields like runtimeId

Bootstrap doesn't throw on any of these. It logs them and proceeds with the records that read clean.

hivecast seed reports the same picture honestly (hivecast.mjs:511-554):

hivecast seed --home /tmp/matrix-home

{
  "ok": true,
  "home": "/tmp/matrix-home",
  "seeded": true,
  "runtimeRecords": {
    "healthy": 9,
    "corrupt": [
      {
        "runtimeId": "RUNTIME-HOST-LOCAL-XXXX-HOST-CONTROL",
        "path": "/tmp/matrix-home/runtimes/.../runtime.json",
        "kind": "empty",
        "size": 0
      }
    ]
  }
}

A subsequent hivecast start will self-heal the empty record by re-running bootstrap-default-runtimes, which calls host-service up for the missing target — and up overwrites a corrupt record with a fresh one.

Stale-record detection

A record may say running while the process is actually dead, or worse, while the pid has been reused by an unrelated process. runtimeRecordStaleReason (hivecast.mjs:1200-1230) checks:

• pid is set and alive (process.kill(pid, 0))
• metadata.command matches the actual cmdline[0] of the running process
• metadata.args is a prefix of the actual cmdline[1..]
• metadata.cwd matches the actual /proc/<pid>/cwd

If any check fails, the record is stale. hivecast doctor flags it; repairDeadRuntimeRecords (hivecast.mjs:859-881) rewrites it as failed with metadata.staleReason so the next start treats it as needing a fresh spawn.

Note: cmdline and cwd checks only work on Linux. On macOS / Windows, only the pid-alive check applies. Cross-platform parity is target state.

Where records come from

• Default Host runtimes — written by bootstrap-default-runtimes after hivecast install or system reboot. Each one has startup: auto, restart: always.
• User-initiated runtimes — written by hivecast up <pkg> with whatever flags the user passed.
• Internal runtimes — written by system.runtimes.runtimes.register ops from host-control heartbeat (e.g. for browser tabs that mount actors).

See also

• Logs — read alongside <runtime-id>.stderr.log for full context.
• Reference → Filesystem layout — the runtimes/ directory in context.
• CLI → hivecast up / down — the commands that create / remove records.
• Troubleshooting → Package failed to start — when records get stuck in failed.

Source: projects/matrix-3/packages/host-service/src/types.ts lines 79-115 (schema); projects/matrix-3/packages/host-service/src/host-state-store.ts lines 90-142 (read/write); projects/matrix-3/packages/hivecast/bin/hivecast.mjs lines 511-554 (scanRuntimeRecordsHealth) and 1200-1273 (stale detection); WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/.