Appearance
Device health
A paired Device is healthy when its most recent heartbeat is within the heartbeat TTL and its NATS user JWT is unrevoked. This page covers the heartbeat protocol, the online/offline computation, and credential refresh.
Heartbeat protocol
A paired Device's host.control periodically POSTs to the platform's HTTP gateway:
POST https://hivecast.ai/_auth/host-link/heartbeat
Authorization: Bearer <heartbeat-token>
Content-Type: application/json
{ "hostLinkId": "<id>", "hostId": "<install-id>", ... }The heartbeat token is issued at pairing exchange (rotateHeartbeatToken in HostLinkStore) and stored as heartbeatTokenHash (HMAC) on the Host Link record. The plaintext token lives only on the Device.
The gateway calls auth.hostLink.verifyHeartbeat:
ts
auth.hostLink.verifyHeartbeat({ token, hostLinkId, hostId, principalId })
// Returns { ok: true, hostLink } if the token matches the stored hash.
// Returns { ok: false, error: 'Invalid Host-link heartbeat token' } otherwise.Source: projects/matrix-3/packages/system-auth/src/index.ts:861-879.
On verify, lastRefreshedAt is updated. system.devices reads lastRefreshedAt plus heartbeatTokenIssuedAt to compute the live status.
Online vs offline
A Device is "online" if lastRefreshedAt > now - heartbeatTtl. The TTL today is configured via the heartbeat schedule (a Device should heartbeat every ~30 seconds; the TTL allowance is a few minutes).
A Device is "offline" if it has a Host Link but no fresh heartbeat. Offline Devices remain in devices.list (they are linked, just not currently reachable).
A Device is "revoked" if link.status === 'revoked'. Revoked Devices are excluded from devices.list by default; pass includeRevoked: true to include them.
Heartbeat schedule
The Device-side heartbeat schedule is owned by the Host's host.control runtime. The current cadence is short (seconds, not minutes) to support a responsive online/offline UI on the dashboard.
Heartbeat traffic is small — a single POST with a short JSON body. It is the cheapest indicator of Device liveness available, much cheaper than the alternatives (probing every runtime).
Credential refresh
Per HIVECAST-DEVICE-ENROLLMENT-SPEC.md Step 8 (proven on richard-refresh-proof-20260501-065817), refresh issues a fresh device-scoped NATS JWT without re-pairing.
POST https://hivecast.ai/_auth/host-link/credentials/refresh
Authorization: Bearer <heartbeat-token>Calls auth.hostLink.credentials.refresh. The op (system-auth/src/index.ts:822-859):
- Verifies the heartbeat token / Host Link.
- Fetches the principal's NATS account.
- Issues a fresh device-scoped JWT (
createDeviceScopedNatsCredentials). - Records the new user public key on the link.
- Old user public keys are revoked through the account JWT.
- NATS hub reloads.
- Returns the new credentials to the Device.
The Device persists the new credentials. The CLI command mx refresh-credentials --home <matrix-home> does the call from a Device.
Refresh is the answer to "rotate this Device's secret." It is also the answer to "the Device's JWT expired" — the device-scoped JWT has a 24-hour TTL (DEVICE_NATS_CREDENTIAL_TTL_SECONDS = 86_400), so Devices refresh at least daily.
What revokes during refresh
The refresh response includes previousNatsUserPublicKeys — the keys that just got revoked through the account JWT update. After refresh:
- The old user JWT's public key is in the account revocation list.
- The NATS hub reloads.
- The old credentials no longer authenticate (proven in the live test).
This means: even a stolen Device JWT becomes useless after the next refresh on the legitimate Device. It is a soft rotation — for hard rotation under suspected compromise, revoke the Host Link entirely.
Health on the dashboard
Per P1.22 progress notes, the dashboard separates bus-auth health from actor health:
- Bus-auth probe —
/api/nats-jwt?purpose=probereturns whether the browser's session can mint a NATS JWT. This is one health signal. - Actor health probe — request/reply against
system.runtimes runtimes.list,host.control status,system.devices devices.list, and per-runtimesystem.runtimes.<runtimeId>.controlruntime.health. These are the deeper health signals.
A Device card on the dashboard rolls these into one status pill — Online, Offline, Degraded, Revoked. The route-plan diagnostics show why.
Failure modes
- Stale heartbeat token. If the Device's
<host-home>/credentials/hivecast-link.jsonis missing the heartbeat token, every heartbeat returns 401. Repair: re-pair the Device. - Hub reload failed. If
auth.hostLink.credentials.refreshissued a fresh JWT but the NATS hub did not reload, both old and new keys are valid temporarily. Repair: bounce the hub. - Clock skew. If the Device's clock is significantly behind, JWT expiry checks may reject otherwise-valid tokens. Repair: NTP.
system.deviceslag. Ifhost.controlheartbeats butsystem.devices.devices.listdoes not show the Device, a registration step did not propagate. See Reference: Operational runbooks.
Live proof
Per WORKSTREAMS/product-launch/HIVECAST-DEVICE-ENROLLMENT-SPEC.md:
- Refresh proof (
richard-refresh-proof-20260501-065817): old credentials connected before refresh; no-token refresh denied; CLI refresh returnedrefreshed:true; old NATS user rejected afterward; new NATS user connected; persisted credential/link files contained no forbiddenaccountSeed,deviceName, or device-meaningdisplayName.
Target state
- Heartbeat over NATS instead of HTTP (cheaper, fits the bus-native model).
- Per-runtime health rollup into the Device card automatically.
- Health alerts — a Device that goes from online to offline triggers a notification.
See also
- Device inventory — where the heartbeat data ends up.
- Disconnect / unlink — when revoke is the right answer.
- Edge / Device Management / Health — Edge-side health view.
- Reference: Actor surfaces — full op list.
Source:
projects/matrix-3/packages/system-auth/src/index.ts:822-879.projects/matrix-3/packages/system-gateway-http/src/host-platform-http-surface.tsfor the HTTP routes.