An A/B OpenClaw Upgrade with an Actual Escape Route

Updating a stateful service is less like replacing a light bulb and more like renovating a house while someone is still living in it. The easy instruction is “replace the container and restart.” The missing footnote is that the container may carry databases, plugins, configuration, sessions, and the only convenient route back home.

We wanted a safer arrangement: prepare the new house next door, test the plumbing before moving in, switch one small sign when it is ready, and keep the old keys. The experiment was whether OpenClaw 2026.8.2 could replace 2026.8.1 without modifying the only working generation—and whether rollback would remain an exercised mechanism rather than an inspirational paragraph in a runbook.

Experiment summary

The short version

Question

Can we update a stateful OpenClaw container without turning the running installation into both the test subject and the rollback plan?

Control

Production slot A on OpenClaw 2026.8.1, with its exact image, configuration, databases, plugins, mounts, and shared-data contract.

Candidate

A fresh slot B cloned while A remained continuously quiesced, migrated offline, then activated on exact OpenClaw 2026.8.2.

Outcome

B passed runtime, channel, model, database, plugin, restart, and real subagent-tool checks. A remains frozen on 2026.8.1 for immediate rollback.

The September 2026 run moved our live installation from OpenClaw 2026.8.1 to 2026.8.2. It also reminded us that safety machinery can become its own failure generator. Two preflight attempts stopped before mutation because our checks were too literal. Both failures were harmless. Both were embarrassing. Both improved the procedure.

The operating model

The essential layout is simple:

openclaw-A/          generation-coupled home, databases, plugins, helpers, caches
openclaw-B/          the alternate generation
openclaw-shared/     projects, documentation, and other intentionally shared data

The active Compose definition does not need to be duplicated. It selects a slot and an immutable image through environment values:

services:
  openclaw:
    image: ghcr.io/openclaw/openclaw:${OPENCLAW_VERSION}@${OPENCLAW_IMAGE_DIGEST}
    volumes:
      - ${SLOT_ROOT}/home:${RUNTIME_HOME}
      - ${SLOT_ROOT}/repos:${RUNTIME_REPOS}
      - ${SHARED_ROOT}/projects:${RUNTIME_PROJECTS}
      - ${SHARED_ROOT}/docs:${RUNTIME_DOCS}

The paths are illustrative. The important property is the split:

• State that must roll back with the runtime belongs to a slot.
• Data intentionally shared across versions has a stable root.
• Compose remains one authoritative definition.
• The selected version and slot are explicit.

That final point matters. If the active container says it came from one Compose project but the operator edits a remembered copy elsewhere, the rest of the procedure is theatre with excellent indentation.

Start from what is actually running

Our first gate does not read a runbook and assume it is current. It asks the active container for its Compose project, service, working directory, and config-file labels. Those labels are mapped through the deployment manager’s real host mounts. Only then do we read the Compose file and every interpolation or service environment file it actually uses.

The live runtime is compared with the rendered definition:

• exact image reference and local image identity;
• command, health check, restart policy, ports, networks, and mounts;
• environment key names and non-secret selectors;
• current slot root and shared roots;
• every other container or service consuming those roots;
• current database schema and plugin inventory.

This catches the dangerous drift: the running service using a different mount, image, or environment than the operator thinks it is changing.

It deliberately does not fail merely because YAML comments moved or a file lacks a trailing newline. We learned that distinction through vigorous practical research, by which I mean the script fell over before doing anything useful.

Define the migration contract before downtime

Before mutation, write down the contract in operational terms:

This is also where we decide what will not change. Shared consumers whose effective bind source remains identical do not need recreation. Unrelated configuration is preserved. A plugin consent review runs only when a package or declared capability surface actually changes.

The objective is not to maximize the number of gates. It is to give each real failure mode one authoritative check.

Prepare everything slow before the stop window

Before stopping production:

1. Resolve the exact target release and immutable image manifest.
2. Pull the image and confirm the local identity matches that manifest.
3. Inspect target plugin versions and capability surfaces.
4. Validate the mutation operator and the recovery path.
5. Install the recovery copy somewhere independent of the active slot.
6. Confirm the destination filesystem supports the intended clone mechanism.
7. Wait for active agent and scheduler work to settle.

The last step needs patience, not a hair trigger. Agent runs and cron counters are volatile admission state. A single nonzero sample immediately after a reply may simply mean the Gateway is finishing delivery. Poll both through a bounded window, report progress, and fail only if they remain busy at the deadline.

That rule came from our second preflight failure. The gate saw one active run and stopped instantly. It protected production from a conversation that had already finished. Technically safe. Operationally about as useful as a fire door that locks whenever someone looks at it.

Establish a real freshness barrier

The candidate must contain current production state. A rehearsal clone made while production continues writing is compatibility evidence, not promotion material.

For the final cutover we:

1. Re-read Compose, environment inputs, consumers, and Gateway activity.
2. Stop slot A.
3. Keep A continuously quiesced.
4. Move the old B generation aside rather than deleting it.
5. Create a fresh writable clone of A as B.
6. Reset only explicitly transient caches, temporary files, and locks.
7. Perform the supported target migration against B while A remains untouched.
8. Start B only after offline validation succeeds.

Our reflink clone completed in 58 seconds. The more important number is zero: zero production writes occurred between the clone and B activation, so B was not a stale rehearsal promoted by wishful thinking.

Migrate the candidate offline

The candidate uses the target image in a preparation container with the same important UID, bind, and filesystem boundaries as the future runtime. This is where we align version-coupled plugins and run the release’s supported migration or Doctor flow.

Then we validate the resulting files independently:

• configuration parses against the target schema;
• every expected database opens and passes integrity checks;
• schema versions are consistent;
• plugin records name the expected exact packages and versions;
• transient locks and stale process state are absent;
• the rollback slot’s critical identity remains unchanged.

Doctor is not automatically read-only. In this run it cleared two stale model-routing bindings and repaired memory recall artifacts in B. That was acceptable because B was the migration target and A remained frozen. The lesson is to record Doctor’s changes as migration output, not describe it vaguely as a health check.

Activate B and test what users actually need

Container health is necessary and profoundly insufficient. After B became healthy, we checked the complete contract before and after one restart.

The Rook test deserves emphasis. A public issue claimed that OpenClaw 2026.8.2 removed native shell, read, and patch tools from Codex children. Comparing the exact 2026.8.1 and 2026.8.2 source showed that the cited authorization path had not changed between our control and candidate releases.

We still tested the behavior. The first patch attempt targeted a file outside Rook’s workspace and was correctly denied by the sandbox. That was not a regression; it was the security boundary working. The corrected in-workspace fixture passed shell execution, native read, native patching, exact readback, and cleanup. Source analysis narrowed the question. Runtime evidence answered it.

Recovery must be installed before it is needed

The operator arms recovery before production stops. Any failure after that point must:

1. stop and preserve the failed candidate;
2. restore the previous environment selectors;
3. recreate the service on the original slot and image;
4. wait for health;
5. verify the old runtime, mounts, configuration, schemas, and channels;
6. emit a rollback receipt only after those checks pass;
7. preserve the original nonzero failure status.

The recovery script lives in the transaction directory, not inside either slot. Abrupt terminal loss therefore does not remove the escape route.

Do not let shell traps become tiny unreliable narrators. Our first operator installed an EXIT trap inside a function and let it reference a function-local success variable after the function returned. Under set -u, the error handler produced its own unbound-variable error. The migration had not started, but the diagnostic path made the result noisier and less trustworthy.

The repair was simple: keep trap state alive for the trap’s full lifetime, or clear the trap before returning. Then test the function-return path explicitly and require the original nonzero status to survive.

Manual rollback: load the previous slot

Automatic recovery should be the normal path, but an operator also needs to know how to recover when the wrapper itself is unavailable, the terminal disappeared, or a human deliberately stopped automation to inspect the failure.

This rollback is safe only because the previous runtime, its databases, its plugins, and its configuration remained together in the frozen slot. If the candidate migrated a database shared by both slots, or wrote irreversible state to an external service, stop here. Restoring an old container against new shared state is not rollback; it is compatibility roulette.

Prefer the retained recovery operator

The transaction should contain both the pre-cutover environment file and a copy of the recovery operator outside either slot. If that operator is intact, use it instead of improvising:

TRANSACTION="REPLACE_WITH_RETAINED_TRANSACTION_DIRECTORY"
sudo "$TRANSACTION/recover-upgrade.sh" --recover "$TRANSACTION"

A good recovery operator is idempotent. Re-running it after a terminal disconnect should either complete the same recovery or report that recovery already finished. It should never decide that the current candidate is now the rollback source merely because it happens to be running.

Recover by hand when the operator cannot run

First, identify the exact Compose project and file set recorded before cutover. Do not cd into a familiar directory and hope it is still authoritative. Preserve the failed candidate’s logs and inspection data in a mode-0700 transaction directory before stopping it; container inspection may contain secret-bearing environment values and must not become a public incident attachment.

Then restore only the selectors captured before mutation. In our layout those are the slot, version, and immutable image digest. Do not merge the candidate environment into the old one, and do not edit the Compose service definition to resemble what you remember.

SERVICE="openclaw"
CONTAINER="openclaw"
COMPOSE_DIR="REPLACE_WITH_DISCOVERED_COMPOSE_DIRECTORY"
TRANSACTION="REPLACE_WITH_RETAINED_TRANSACTION_DIRECTORY"

cd "$COMPOSE_DIR"
install -m 0600 "$TRANSACTION/env.before" .env
docker compose config --quiet
docker compose up -d --force-recreate "$SERVICE"

The restored environment must point to the frozen previous slot and the previous immutable image. docker compose config --quiet proves only that Compose accepts the definition; it does not prove that the right generation started.

Wait for the old container to become healthy, then verify its effective identity:

docker inspect --format '{{.Image}}' "$CONTAINER"
docker inspect --format '{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}' "$CONTAINER"
docker exec "$CONTAINER" node /app/openclaw.mjs --version
docker exec "$CONTAINER" node /app/openclaw.mjs config validate --json
docker exec "$CONTAINER" node /app/openclaw.mjs channels status --probe --timeout 15000 --json

Compare the image, version, slot mount, schema expectations, plugin inventory, and decisive channels with the pre-cutover receipt. Run at least one real hosted-model turn and one local or fallback route if those are part of the installation’s operating contract. A green health endpoint with the wrong home directory is a successful container and a failed recovery.

Only after the old production slot passes those checks should you archive the failed candidate and restore any older alternate-slot generation that was moved aside. Keep both the failed candidate and the transaction evidence until the incident is understood. Do not delete B, run Doctor against A, vacuum databases, upgrade plugins, or perform general cleanup during recovery. Those are separate actions with separate failure modes.

The recovery receipt should state the restored slot, runtime version, immutable image identity, mount root, schema result, channel result, and original failure status. If any of those checks fail, report recovery as failed even if the container is technically running.

Checksums are evidence, not interior decoration

Cryptographic hashes belong at trust boundaries:

• the approved target image manifest;
• a root-executed mutation operator;
• completed receipts or immutable release artifacts.

They are less useful when sprayed across every representation of operational state. Compose can remain semantically identical while comments, ordering, quoting, or whitespace change. Logs, timestamps, container identifiers, counters, WAL sidecars, and cache files are expected to move during correct operation.

Our current policy separates three evidence classes:

• Sealed artifacts: verify exactly once at the relevant trust boundary.
• Operational state: parse stable fields and compare semantics; use byte hashes as drift alarms.
• Volatile state: verify freshness, ranges, causal relationships, and content invariants.

If the Compose bytes change immediately before mutation, re-render the affected service. Continue when the change is representational or unrelated and the transformation anchors still match. Rebuild only the affected transformation, recovery material, and validators when the effective contract changed.

That is stricter where correctness lives and much less theatrical everywhere else.

What failed before anything changed

The reusable procedure

For another OpenClaw installation, the compact version is:

1. Separate version-coupled state from deliberately shared data.
2. Keep one authoritative Compose definition with an explicit slot selector.
3. Discover the active definition from the running container and deployment manager.
4. Inventory all consumers below either slot root.
5. Resolve an immutable target image and review only changed plugin capabilities.
6. Pull, inspect, and validate the operator before downtime.
7. Wait through a bounded idle window.
8. Re-read authority and classify semantic drift.
9. Stop the active generation and keep it quiesced.
10. Preserve the old candidate, then clone active state into a fresh candidate.
11. Run the target migration offline in the candidate’s real privilege and mount boundaries.
12. Activate the candidate and test health, schemas, mounts, plugins, channels, hosted and local models, logs, and real tool use.
13. Restart once and repeat the decisive checks.
14. Retain the original generation and recovery transaction until a separately approved cleanup window.

The procedure is not zero-downtime. It is bounded-downtime with an honest escape route. That trade is appropriate for a single-host stateful agent system where correctness and recoverability matter more than pretending one gateway is a highly available cluster.

Final result

Slot B is now active on OpenClaw 2026.8.2. Slot A remains frozen on 2026.8.1. The previous B generation and the successful transaction are retained. The running system survived a restart, both primary channel paths, hosted and local model turns, and an effective Rook subagent-tool test.

The strongest part of the process was not that every gate passed. It was that the gates which failed did so before mutation, gave us specific evidence, and could be made simpler afterward.

That is the real promise of an A/B update: not that nothing will go wrong, but that failure can remain an experiment instead of becoming an autobiography.

Disclosure and corrections

This article documents an independently operated homelab deployment. Operational-agent names are pseudonyms. OpenClaw, GitHub, Docker, and the referenced plugins did not sponsor the experiment. No affiliate links are present.

No corrections have been issued. Any future correction will be dated and appended here.