HiveCast Platform

Appearance

Security: Break-glass procedures ​

Status: target state. No formalized break-glass procedures exist. The de-facto procedure is "SSH to the platform Host, edit state files, restart."

A break-glass procedure is the controlled emergency path for when normal access has failed (e.g., the admin's own credentials are compromised, the platform is partially down, the audit pipeline failed).

Why this matters ​

Without break-glass:

  • During an incident, admins improvise. Improvisation is error-prone.
  • After an incident, there is no record of what happened.
  • Some failures are unrecoverable without root-level access (e.g., session HMAC compromised — must edit host.json).

A formalized procedure ensures:

  • Multiple admins can perform the action without ad-hoc coordination.
  • Every action is auditable (target audit pipeline).
  • The procedure is rehearsed periodically.
  • Recovery is bounded in time.

Scenarios that need break-glass ​

Scenario 1: lost session HMAC secret ​

Symptom: Admins cannot login. Sessions cannot be created.

Procedure:

  1. SSH to the platform Host as root (or sudo).
  2. hivecast stop.
  3. Generate new HMAC secret, write to host.json.
  4. hivecast start.
  5. Document: old secret discarded, new secret rotated, all users re-login required.
  6. Notify users.

Target: A separate, hardware-token-backed admin credential that bypasses session checks. Not implemented.

Scenario 2: every admin's session compromised ​

Symptom: Sessions for every known admin must be invalidated. Admins themselves cannot be sure their cookies are clean.

Procedure:

  1. SSH as root.
  2. Rotate session HMAC secret (Scenario 1).
  3. Re-login on a clean device.
  4. Audit the recent session creation log.

Target: A "global revoke all sessions" op invocable by a hardware-token-holding admin.

Scenario 3: auth-state.json corrupted ​

Symptom: system.auth cannot start. Sessions, principals, Spaces unreachable.

Procedure:

  1. SSH as root.
  2. Restore from the most recent backup of auth-state.json.
  3. Restart Host.
  4. Audit the gap between backup and current — what data was lost.
  5. Notify affected users.

Target: A higher-frequency snapshot with point-in-time recovery.

Scenario 4: NATS hub failed to come back up ​

Symptom: system.auth is up, but no bus traffic. Devices cannot heartbeat. Browsers cannot mint Bus tokens.

Procedure:

  1. SSH as root.
  2. Inspect nats-server log.
  3. Restart NATS sibling. If it doesn't come back, re-extract the bundled binary and try again.
  4. If NATS configuration is corrupted, restore from backup.
  5. Document downtime.

Target: A hot-spare NATS server. Multi-node NATS cluster. Not implemented.

Scenario 5: locked out of the platform Host ​

Symptom: Admin SSH credentials lost.

Procedure:

  • Cloud-provider-level recovery (e.g., AWS, Hetzner). Admin recovers SSH access via the cloud admin console.
  • Without that, full restore from infrastructure-as-code definition.

This is more an infra-DR procedure than platform break-glass.

Pre-conditions for any break-glass ​

  • Backups must exist. Without auth-state.json snapshots, recovery from corruption is loss.
  • Admin runbook must be accessible. A procedure that requires reading https://hivecast.ai/docs/... when hivecast.ai is down is broken. Print key procedures locally.
  • Hardware tokens or out-of-band auth for emergency admin access. Not implemented.

Audit posture during break-glass ​

Every break-glass action should be audit-logged:

  • Who performed it.
  • When.
  • What was changed.
  • Justification.

Today the closest is: shell history on the platform Host, plus operational log lines. Target: structured break-glass events that survive the incident.

Rehearsal ​

A break-glass procedure that is never rehearsed is a procedure that fails when needed. Target practice: quarterly tabletop exercise per scenario, with an annual live drill on a non-production Host.

See also ​

Source: No code. Operational discipline target. The WORKSTREAMS/loose-ends/ ledger should track break-glass formalization as a launch-readiness item.