Appearance
Heartbeats
A Device's presence in system.devices is driven by heartbeats. The Device's own host.control actor posts them at startup and on a fixed interval, both to the local system.devices actor (over the bus) and, if the Device is cloud-linked, to a cloud HTTP endpoint.
Two heartbeat targets
text
host.control
│
├──► local bus: system.devices.devices.register
│ + system.devices.devices.runtimes.register
│
└──► cloud HTTPS: POST <cloudUrl>/_auth/host-link/heartbeat
Authorization: Bearer <hostLinkToken>A linked Device posts to BOTH targets every interval. A local-owner Host (not linked) posts only to the local bus.
Frequency and timeouts
Constants live in HostControlActor.ts:
| Constant | Value | Use |
|---|---|---|
DEVICE_HEARTBEAT_INTERVAL_MS | (interval, fixed in code) | timer firing rate |
DEVICE_HEARTBEAT_TIMEOUT_MS | bus call timeout | per-op RequestReply.execute() timeout |
DEVICE_HEARTBEAT_TTL_MS | passed in heartbeatTtlMs | how long the receiver should treat this Device as online |
On the receiver side:
| Constant | Value | Use |
|---|---|---|
DEFAULT_HEARTBEAT_TTL_MS | 45_000 | fallback if heartbeat omits its own TTL |
PRUNE_INTERVAL_MS | 15_000 | how often _pruneStaleDevices() runs |
Local heartbeat path
HostControlActor._publishDeviceHeartbeat() runs at startup and on interval. It:
- Reads
<host-home>/credentials/hivecast-link.jsonviareadHiveCastHostLink(). If the link is missing or incomplete (nohostId,principalId, orauthorityRoot), it does nothing — local-only Hosts skip heartbeating. - Reads runtime records via
_listRuntimeRecordsForHeartbeat(link). Only runtimes whoseruntimeWireRootmatches the Host Link'sauthorityRootare advertised. - (If cloud-linked) calls
postCloudDeviceHeartbeat(). - Calls
_registerLocalDeviceHeartbeat()which firesdevices.registerand onedevices.runtimes.registerper runtime.
The Device payload (from deviceHeartbeatPayload() at lines 393-413) is:
ts
{
deviceId: link.hostId,
hostLinkId?: link.hostLinkId,
principalId: link.principalId,
hostName?: link.hostName,
deviceSlug?: link.deviceSlug,
spaceId?: link.spaceId,
linkStatus: 'active',
authorityRoot: link.authorityRoot,
publicNamespace?: link.publicNamespace,
routeKey?: link.routeKey,
heartbeatTtlMs: DEVICE_HEARTBEAT_TTL_MS,
metadata: {
source: 'host-control',
reason,
cloudUrl?: link.cloudUrl,
deviceRegistryRoot?: link.deviceRegistryRoot,
},
}reason is 'startup' on the first heartbeat and 'interval' thereafter.
Cloud heartbeat path
postCloudDeviceHeartbeat() is the cloud-side post. It:
ts
// projects/matrix-3/packages/host-control/src/HostControlActor.ts:474-492
const endpoint = new URL('/_auth/host-link/heartbeat', link.cloudUrl);
const response = await fetch(endpoint, {
method: 'POST',
headers: {
authorization: `Bearer ${link.hostLinkToken}`,
'content-type': 'application/json',
},
body: JSON.stringify({
hostLinkId: link.hostLinkId,
hostId: link.hostId,
principalId: link.principalId,
device: deviceHeartbeatPayload(link, reason),
runtimes: runtimes
.filter((runtime) => readString(runtime.runtimeId).length > 0)
.map((runtime) => runtimeHeartbeatPayload(link, runtime)),
}),
});The cloud gateway calls auth.hostLink.verifyHeartbeat({ token, hostLinkId, hostId, principalId }) to validate the bearer. If the token does not match the SHA-256 hash on the active Host Link, or the link is revoked, the request is rejected.
After successful verification, the gateway updates system.devices with the device + runtime payloads, and updates lastRefreshedAt / updatedAt on the Host Link via HostLinkStore.refresh() / verifyHeartbeatToken().
What identity comes from where
Per CLAUDE.md Rule 5 ("Identity from transport metadata, NEVER from payload"):
| Identity field | Source on cloud-side heartbeat |
|---|---|
principalId | derived from the bearer token's verified Host Link record, NOT trusted from request body |
authorityRoot | derived from the bearer token's verified Host Link record |
spaceId | derived from the bearer token's verified Host Link record |
hostId / hostLinkId | matched against the body for shape, but the verified Host Link is canonical |
Body fields that disagree with the verified link are rejected. This is the property that scenarios A-E in WORKSTREAMS/loose-ends/items/P1.17-device-heartbeat-auth-proof.md tests.
Failure modes
| Symptom | Cause | Fix |
|---|---|---|
device.heartbeat.failed { reason: 'register-failed' } event | system.devices returned { ok: false } from devices.register. Most common cause: the previous record's identity scope disagrees with the new payload (see Ownership). | Inspect the existing Device record. If it's a stale leftover, remove it via the appropriate mutation op or re-pair the Device. |
device.heartbeat.failed { reason: '<error message>' } event | Either readHiveCastHostLink() failed (corrupt or missing hivecast-link.json) or the cloud heartbeat HTTP request threw | Check <host-home>/credentials/hivecast-link.json is well-formed. Check the cloud is reachable. |
Cloud device heartbeat failed: HTTP 401 | The bearer token is wrong. Either rotate happened cloud-side without the Device picking up the new value, or the link was revoked. | hivecast link-status to confirm; re-pair if revoked. |
Cloud device heartbeat failed: HTTP 403 | The link exists but is revoked, or the body identity does not match the bearer-verified identity. | Per CLAUDE.md Rule 5 the server uses the bearer-derived identity. If the body says a different principalId, the request is rejected. Investigate the heartbeat caller. |
What heartbeat does NOT do
- It does NOT push individual op invocations. Heartbeats only update presence and runtime metadata.
- It does NOT mint NATS credentials. Credential rotation is a separate op (
auth.hostLink.credentials.refresh). - It does NOT re-create the Host Link. If the cloud-side row is missing (e.g. revoked + cleaned up), heartbeats fail and require re-pairing.
See also
- Online / offline status — how heartbeat staleness translates to status changes.
- Ownership — which body fields are trusted vs authoritative.
- Operations / Troubleshoot — runbook for heartbeat failures.
Source:
projects/matrix-3/packages/host-control/src/HostControlActor.ts(_publishDeviceHeartbeat(),deviceHeartbeatPayload(),postCloudDeviceHeartbeat());projects/matrix-3/packages/system/src/SystemDevicesActor.ts(onDevicesRegister()accepts the heartbeat).