Rollouts

Status: target state. No rollout orchestrator exists today. This page documents the intended model.

Today's reality

A "rollout" today is "rsync the new wrapper to every Device, restart in some order, hope it works."

# What an operator does manually
for host in laptop-01 laptop-02 laptop-03; do
  ssh $host "sudo apt upgrade hivecast=0.1.25"
  ssh $host "hivecast stop && hivecast start"
  # Hope nothing broke. Move to next.
done

This is fine for a small fleet operated by one person. It does not scale and provides no:

• Health-gating (don't continue if Device 1 became unhealthy).
• Phased rollout (canary first, then 10%, then full).
• Automatic rollback (revert if error rate spikes).
• Audit trail.
• Cancel.

Target rollout model

A rollout would be:

rollout:
  profile: production
  fromRevision: 14
  toRevision: 15
  strategy: phased
  phases:
    - canary: 1 device
      gateMs: 600000      # 10 min observe before next phase
      healthGate:
        - all-runtimes-healthy
        - error-rate < 0.01
    - bulk: 25%
      gateMs: 600000
    - bulk: remainder
  rollback:
    on: healthGate-fail

The orchestrator would:

1. Snapshot the current system.devices membership for the profile's scope.
2. Apply the new revision to the canary slice via host.control profile.apply.
3. Wait gateMs, evaluate healthGate against system.devices and per-runtime health.
4. If healthy, advance to the next phase. If not, roll back.
5. Continue through phases until done.
6. Emit a structured audit record per phase transition.

Why orchestration is hard

Three things make rollout orchestration non-trivial:

• Heterogeneous Devices. A Device might be offline mid-rollout. Skip it (and roll forward when it returns) or block the phase?
• Asymmetric latency. A Device with slow disk takes longer to apply. Health gates need timeout handling.
• Cross-runtime dependencies. Director depends on inference; deploying inference v2 first might break Director temporarily.

These are solved problems in container orchestrators (Kubernetes, Nomad). The Matrix-native primitives — host.control profile.apply, system.devices health, system.runtimes introspection — are sufficient to build it. The orchestrator package itself does not exist.

Manual phased rollout today

Operators wanting some safety:

# Phase 1: canary
ssh laptop-01 "sudo apt upgrade hivecast=0.1.25 && hivecast restart"
matrix invoke system.devices devices.list '{"deviceSlug":"laptop-01"}'
# Check status: online, runtimes healthy
# Wait, observe error logs

# Phase 2: 25%
for host in laptop-02 laptop-03; do
  ssh $host "sudo apt upgrade hivecast=0.1.25 && hivecast restart"
done
# Repeat checks

# Phase 3: rest

This is the pragmatic workaround.

Rollback during rollout

If a rollout has reached phase 2 and phase 2 health-gate fails, the orchestrator should:

• Stop advancing.
• Apply the previous revision back to the already-upgraded slice.
• Mark the rollout as failed.

Without an orchestrator, a manual rollback is "downgrade the wrapper on every already-upgraded Device." That is fine if you have inventory tracking; otherwise you risk leaving Devices on the bad version.

Loose ends

There is no WORKSTREAMS/rollouts/ directory yet. The work is anchored in WORKSTREAMS/docker-npm-parity/ as a downstream consumer of the registry-pull and profile work.

See also

• Deployment profiles — what gets rolled out.
• Package versions — version transitions.
• Rollback — undoing a rollout.

Source: WORKSTREAMS/docker-npm-parity/. No code today.