Operations: Resolve incident

This page is a playbook for high-impact platform incidents. Each incident type has a fast-path response and a follow-up.

Incident type 1: principal account compromise

Signal: Suspicious activity on a principal's Devices, unexpected pair attempts, suspicious sessions.

Fast path:

1. Identify the principal (auth.principal.list, or check audit logs for the suspicious activity).
2. Revoke all sessions for the principal:

   # No bulk op today. The HMAC-signed sessions are 7-day. Options:
   # (a) Rotate session HMAC secret (invalidates ALL users' sessions, not just one).
   # (b) Add each session jti to revoked-sessions manually.
   # The auth-state.json holds .revokedSessions[].

3. Revoke all the principal's Host Links:

   matrix invoke system.auth auth.hostLink.list '{"principalId":"p_xxx"}'
   # For each id, revoke

4. Suspend the principal (manual: edit auth-state.json principals[].status = 'suspended').
5. Restart Host so the suspended status takes effect for any cached identity.

Follow-up:

• Audit log review for the affected window.
• User notification (out-of-band, since their account is suspended).
• Re-onboard via password/credential reset (target — not implemented, would currently mean creating a new principal).

Incident type 2: fleet-wide outage

Signal: Multiple Devices report offline; multiple app routes return errors; bus traffic stalls.

Fast path:

1. Triage the platform Host:

   curl -fsS http://localhost:3100/healthz
   matrix invoke host.control status '{}' --home <host-home>
   tail -50 <host-home>/logs/runtimes/system-gateway-http.log
   tail -50 <host-home>/logs/runtimes/system-auth.log

2. Triage NATS:

   nats server check --server nats://127.0.0.1:4222

3. If the platform Host is wedged: restart.

   hivecast stop && hivecast start

4. If only specific runtimes are wedged: restart those runtimes, not the whole Host.

Follow-up:

• Identify whether the trigger was a deploy (correlate timing with Operations: Deploy package update).
• If yes, consider rollback (Operations: Roll back package update).
• Capture logs for postmortem.

Incident type 3: credential leak

Signal: A Device's NATS user JWT was found in a public location (paste site, leaked log).

Fast path:

1. Identify the link by NATS user public key:

   matrix invoke system.auth auth.hostLink.list '{"principalId":"<owner>"}'
   # Find the row whose natsUserPublicKeys[] contains the leaked key.

2. Revoke the link:

   matrix invoke system.auth auth.hostLink.revoke '{"hostLinkId":"<id>"}'

3. Verify NATS hub reload — the leaked JWT should be rejected within seconds.
4. Re-pair the Device via the standard flow.

Follow-up:

• Audit how the leak happened (Device-side log? CI artifact? backup file?).
• If the leak might extend to other Devices' credentials, refresh credentials proactively:

  for hostLinkId in <ids>; do
    matrix invoke system.auth auth.hostLink.credentials.refresh "{...}"
  done

Incident type 4: NATS account seed compromise (catastrophic)

Signal: A principal NATS account seed was leaked — every Device under that principal can be impersonated.

Fast path:

1. Rotate the principal NATS account. This invalidates every Device's user JWT under it.
   • Today this requires direct manipulation of auth-state.json credentials[] and the NATS account JWT. There is no auth.nats.account.rotate op.
2. Revoke every Host Link for the principal (already invalidated by the account rotation, but record the revoke for audit).
3. Re-pair every Device. Each re-pair issues a fresh user JWT under the new account.
4. Suspend the principal until re-pair complete (prevent further account use).

Follow-up:

• Investigate seed exposure.
• The principal account seed is supposed to never leave the server (per HIVECAST-DEVICE-ENROLLMENT-SPEC.md); a leak is a server-side process bug.
• Add or strengthen access controls on auth-state.json.

Incident type 5: session HMAC secret compromise

Signal: Anyone could mint valid mx_session JWTs.

Fast path:

1. Generate a new secret (HostSessionService.generateSecret()).
2. Update host.json.
3. Restart the platform Host.
4. All sessions invalidated. Every user must re-login.
5. Audit the leak vector.

Follow-up:

• Any data accessed during the compromise window is suspect.
• Audit log review.
• Post-mortem on secret storage discipline.

Generic incident posture

For all incidents:

• Capture state first. Snapshot <host-home>/state/, <host-home>/logs/, current system.devices list, current auth.principal.list. Before mutation.
• Communicate. Internal comms channel + user-facing status page (target).
• Document. Even ad-hoc incident response benefits from a written timeline.
• Plan recovery. Don't just stop the bleeding; restore service.

See also

• Security: Break-glass procedures — when normal access fails.
• Security: Audit trails — what should have been logged.
• Operations: Disconnect a device — revoke flow.
• Reference: Operational runbooks.

Source: Operational experience captured across WORKSTREAMS/loose-ends/ and WORKSTREAMS/product-launch/.