Operations: Inspect runtime

A runtime in trouble shows up as a degraded app card on the dashboard, a 5xx response from a route, or absence from system.runtimes runtimes.list. This page enumerates the checks an admin runs to triage.

Step 1: identify the runtime

From the dashboard, the bad app card carries a runtimeId. From the CLI:

matrix invoke system.runtimes runtimes.list '{}' --home <host-home>
# Look for a missing runtime, or one with low actorCount, stale lastHeartbeat.

matrix invoke host.control status '{}' --home <host-home>
# Look at runtime records: any in 'failed' state.

Step 2: check the supervisor's view

matrix invoke host.control runtimes.get \
  '{"runtimeId":"<id>"}' --home <host-home>

Returns the runtime record: package, mount, env, port, startup, restart, last start metadata, last error if any.

Step 3: tail the log

tail -100 <host-home>/logs/runtimes/<runtime-id>.log

Look for:

• Start-time errors (immediately after Starting runtime ...).
• Repeated reconnect / auth errors.
• actorLog error severity lines.
• Stack traces.

Step 4: probe via bus

# Runtime should respond to its own control mount
matrix invoke system.runtimes.<runtime-id>.control runtime.health '{}' \
  --home <host-home>

A healthy runtime returns { ok: true, status: 'healthy', ... }. A failure here points to a bus auth issue (the runtime cannot reach system.runtimes) or a process-level issue (it crashed).

Step 5: probe the asset endpoint

For webapp runtimes, fetch the asset:

curl -i http://127.0.0.1:3100/apps/<appName>/
# Expect HTTP 200 with HTML.

A 502 or 503 means the gateway resolved the route but the runtime did not respond. A 404 means route metadata is wrong (webapp.routePrefix mismatch).

Step 6: inspect the route plan

The dashboard's per-app route-plan panel shows:

• URL.
• appName.
• Gateway (system.gateway.http).
• Target authority root.
• Runtime id.
• Package.
• Asset provider (e.g., system.runtimes.<id>.http).

If route-plan data is wrong (runtime id empty, asset provider missing), the runtime did not register correctly with system.devices.

Step 7: check dependencies

A runtime's requiredPackages[] lists its npm-style dependencies. If a dependency is missing or stale:

matrix invoke system.runtimes runtimes.list '{}' --home <host-home>
# Find the runtime row, inspect requiredPackages.

ls <host-home>/.matrix/packages/<dep-name>/
# Confirm the dep exists.

A missing dependency is a packaging bug — either the wrapper bundled wrong, or the runtime declared a dep not in the bundled set.

Step 8: restart strategically

Restart options, least-disruptive first:

# Reload (re-read config without process restart)
matrix invoke system.runtimes runtimes.reload '{"mount":"<mount>"}'

# Restart the runtime
matrix invoke system.runtimes runtimes.restart '{"mount":"<mount>"}'

# Restart the entire supervisor (cycles every runtime)
matrix invoke host.control host.restart '{}'

# Last resort: stop the Host, start it
hivecast stop && hivecast start

Document what fixed it.

Common bugs

Symptom	Likely cause	Fix
Runtime not in runtimes.list after restart	Empty record (zero-byte JSON)	hivecast seed
runtime.health times out	Bus auth failure	Check NATS user JWT validity, refresh credentials
Asset 404	routePrefix mismatch in matrix.json	Verify package install, rebuild
Runtime restarts every minute	Crash loop, restart: always	Tail log, find the panic
actorCount is 0	Runtime started but no actors mounted	Check inventoryMode and inventory[]

See also

• Overview / Runtime hosts — concept.
• Deployments / Runtime hosts — operator workflow.
• Operations: Resolve incident — when runtime issues are part of a wider incident.
• Reference: Operational runbooks.

Source: projects/matrix-3/packages/system-runtimes/src/RuntimeManagerActor.ts for ops. projects/matrix-3/packages/host-control/src/index.ts for supervisor view.