Restart runtime

A runtime's restart policy controls what the Host does when the runtime exits unexpectedly. There are three values:

// projects/matrix-3/packages/host-service/src/types.ts:75-77
type HostRuntimeRestartPolicy = 'never' | 'always' | 'on-failure';

The decision logic is shouldRestartRuntime (matrix-host-service.ts:1215-1227):

function shouldRestartRuntime(record, exitCode, signal): boolean {
  if (record.restart === 'always')     return true;
  if (record.restart === 'on-failure') return !(exitCode === 0 || signal === 'SIGTERM');
  return false; // 'never'
}

Policy	Restart on exit(0)	Restart on exit(N!=0)	Restart on SIGTERM	Restart on SIGKILL
never	no	no	no	no
on-failure	no	yes	no	yes
always	yes	yes	yes	yes

SIGTERM is the supervisor's own graceful-stop signal, so on-failure specifically excludes it: stopping a runtime via matrix down does not trip a respawn.

When the supervisor decides

The decision happens in the child.exit handler at matrix-host-service.ts:464-489:

child.once('exit', (exitCode, signal) => {
  if (this._stoppingRuntimeIds.has(runtimeId)) return; // matrix down in progress, no respawn
  this._runtimeProcesses.delete(runtimeId);
  // ... write final record ...
  if (shouldRestartRuntime(exitRecord, exitCode, signal)) {
    setTimeout(() => {
      void this._restartRuntimeFromRecord(exitRecord, 'runtime-exit');
    }, 1_000);  // 1-second cooldown
  }
});

The 1-second delay (matrix-host-service.ts:485-488) is intentional: it prevents a restart: always + exit(1) runtime from saturating the event loop with respawns. It does not implement back-off — successive crashes will respawn at 1s intervals indefinitely.

_restartRuntimeFromRecord (matrix-host-service.ts:624-645) re-uses the persisted record's command, args, cwd, env, localMounts, and metadata. It refreshes the per-runtime environment file first (picking up any host.json change since spawn), then invokes the same startRuntimeProcess codepath used by matrix up. Restart failures write a restartFailedReason into the record's metadata.

How restart interacts with startup

The two policies are orthogonal:

startup	restart	Behavior
manual	never	one-shot — manual start, manual restart
manual	always	manual start, but respawn forever once started
auto	never	started at Host start; one-shot after that
auto	always	started at Host start; respawn forever — the production default for the gateway, web, and edge runtimes
auto	on-failure	started at Host start; respawn only on crash

The default runtime targets shipped by the wrapper (hivecast.mjs:28-39) all set startup: "auto" but the wrapper does not set restart per target — the operator picks via --restart on explicit matrix up, or restart defaults to "never".

Detecting a runaway restart loop

There is no built-in rate limit at the Host level. systemd's StartLimitBurst=5 and StartLimitIntervalSec=60 apply only to the Host service itself, not to its supervised children. So if the gateway runtime crashes every 1.5 seconds, the Host happily respawns it every 2.5 seconds and journald fills up. The journald cap added by the launch-readiness work (WORKSTREAMS/launch-readiness-atomic-writes-and-bootstrap/) is the disk-safety backstop.

To catch a respawn loop manually:

# Same runtime, many quick startedAt values?
hivecast runtimes \
  | jq '.runtimes[] | select(.runtimeId=="RUNTIME-HOST-LOCAL-ABC-CHAT") | .startedAt'

# stderr should explain why.
tail -n 200 <home>/logs/runtimes/<runtimeId>/stderr.log

If a runtime is in a respawn loop, force restart: never by editing the record on disk and restarting the Host:

runtime_dir=<home>/runtimes/<sanitized-runtimeId>
jq '.restart="never"' "$runtime_dir/runtime.json" > "$runtime_dir/runtime.json.new"
mv "$runtime_dir/runtime.json.new" "$runtime_dir/runtime.json"
hivecast stop --home <home>
hivecast start --home <home>

(Yes, this is the rough edge. A future runtime.update op would close it.)

Restarting an already-running runtime

There is no matrix restart verb. The pattern is:

matrix down @open-matrix/chat
matrix up   @open-matrix/chat --runtime-id ...   # same flags as before

Or, if the runtime had restart: always, send the running process SIGTERM-equivalent through runtime.shutdown and let the supervisor respawn it; that's a bit hacky. The cleaner pattern is matrix down followed by matrix up.

See also

• Start runtime
• Stop runtime
• Shutdown semantics
• Failure modes