HiveCast Platform

Appearance

Operations: Roll back package update ​

For the conceptual model see Deployments / Rollback. This page is the operator's playbook for the present-state manual rollback.

When to roll back ​

Roll back when, after a deploy:

  • App routes start returning errors that were not failing before.
  • Devices flip to offline at higher rate.
  • Inference / chat / director runtimes fail to start.
  • A specific package's behavior regressed (validated on the canary).

Do not roll back for transient symptoms; investigate first. Rollback is disruptive.

Identify the prior good version ​

bash
# Wrapper version your inventory tool says was running before deploy.
# Or check apt history:
grep " install " /var/log/apt/history.log | grep hivecast | tail -5

You need the exact version string (0.1.24, etc.) for apt install.

Roll back per Device ​

bash
# On each Device that received the bad upgrade
sudo apt install hivecast=0.1.<previous>
hivecast stop
hivecast start

# Verify recovery
hivecast --version  # confirms downgrade
matrix invoke host.control status '{}'
matrix invoke system.runtimes runtimes.list '{}'
curl -fsS http://127.0.0.1:3100/healthz

If apt install hivecast=<version> fails because the package version is no longer available:

  • Some Linux distributions purge old versions from the repo.
  • Keep snapshots of prior wrapper builds outside apt for rollback.

Order of rollback ​

Reverse of upgrade order:

  1. Roll back the Devices that show problems first.
  2. If the issue is fleet-wide, roll back canary last (because it's the validation Device).
  3. Verify each Device returns to healthy before moving on.

Rolling back data migrations (target hazard) ​

If a deploy migrated stored state (changed JSON shape, etc.), rollback to a wrapper that doesn't recognize the new shape can crash. Today the platform's stored state is additive — new fields are added, old fields are not removed — so rollback is safe.

As the system grows, migrations will require backward-compat planning:

  • Write the new shape additively.
  • Verify the old code reads the new shape.
  • Only later remove the old shape support.

This is target-state operational discipline.

Rolling back paired Device credentials ​

If the deploy involved a credential format change (e.g., new JWT structure), rollback may invalidate fresh credentials. The fix:

bash
# After rollback, on each Device
matrix invoke system.auth auth.hostLink.credentials.refresh \
  '{"hostLinkId":"<id>","principalId":"p_xxx"}'
# Reissues credentials in the rolled-back format.

Today this is hypothetical; the credential format has been stable.

What rollback does NOT do ​

  • Does not undo new principal records (a user who signed in during the bad-version window stays).
  • Does not undo new Spaces.
  • Does not undo new Host Links.
  • Does not undo audit log entries.

These are append-only state. Rollback restores the code, not the data.

Target: orchestrated rollback ​

Per Deployments / Rollback, the target is one click that:

  • Identifies which Devices got the bad version.
  • Applies the prior revision back to those Devices.
  • Health-gates the rollback (don't continue if Device 1 is now broken).
  • Audits the action.
  • Notifies operators of completion.

This requires the rollout orchestrator (target).

Post-rollback debrief ​

After a rollback, document:

  • What broke and why.
  • Which version revealed the bug.
  • What the fix is for the next deploy.
  • What to add to the rollout health-gate to catch this earlier.

Until automated audit ships, this is operator-discipline.

See also ​

Source: WORKSTREAMS/docker-npm-parity/. Today's procedure is operator experience.