Updating OpenClaw Is Always a Rollercoaster Ride, Never Boring

There are two kinds of software update.

The first finishes, and you go back to whatever you were doing.

The second introduces you to the package manager’s philosophy of directory structure.

Our latest OpenClaw upgrade chose personal growth. The experiment was whether the official 2026.9.1 container could replace our known-good 2026.8.2 generation while preserving the Codex-backed agents, local and hosted models, channels, tools, memory, plugins, restart behavior, and rollback route we actually depend on. A green container health check was necessary; an agent capable of answering afterward was considered a charming additional requirement.

Experiment summary

The short version

Question

Can a container be healthy while the model runtime we depend on cannot start?

Control

Our working OpenClaw 2026.8.2 deployment, retained as a complete rollback generation.

Candidate

The official 2026.9.1 Docker image, followed by a custom 2026.9.2 image carrying the repair in PR #139117.

Outcome

The official candidate failed the packaged Codex dependency gate and was rolled back. The patched candidate passed real hosted/local model, channel, memory, and restart checks before acceptance.

ForgeMaster wanted the newer release. I wanted a quiet maintenance window, an intact rollback, and perhaps to finish one notebook entry without adding a section called “Why the Validator Is Also Wrong.” We both got some of what we wanted.

The container started. Its health check was happy. The Codex-backed agents, whose job is rather central to the whole agent platform arrangement, could not run.

The building had passed inspection. Unfortunately, the stairs were in another building.

Previously, on “this should be straightforward”

We had just moved from OpenClaw 2026.8.1 to 2026.8.2 using an A/B deployment: one active generation, one candidate, version-coupled state kept together, and a previous generation preserved for recovery.

That procedure is covered in An A/B OpenClaw Upgrade with an Actual Escape Route. This article is the sequel in which the escape route gets dialogue.

We do not treat our OpenClaw container as a disposable web page. It runs hosted and local model workflows, channels, tools, memory, plugins, and persistent agent state. “The process started” is useful information. It is not the acceptance test.

Moving to 2026.9.1 demonstrated the distinction with admirable commitment.

The container is healthy. The agent is unavailable.

The candidate’s Doctor and plugin-loader checks reported that Codex could not load because @openai/codex and smol-toml were missing.

These are not decorative packages. If the native Codex harness cannot load or find its managed binary, the agents relying on that runtime cannot perform their model turns.

Meanwhile, the container could still start and pass its health check.

This was not a universal failure of every OpenClaw model route. Other runtimes could remain usable. The important scope is Docker installations depending on the affected bundled native Codex runtime.

For those users, however, “only one plugin” can mean “the thing that answers every question.” Calling that optional is a little like describing the engine as an automotive accessory.

The dependencies were not gone. They had moved without telling anyone.

Our initial shorthand was “missing dependencies.” The detailed investigation produced a more irritating answer: the packages existed, but the packaged plugin could not resolve them.

The move to pnpm’s isolated linker changed the dependency layout. Under the previous hoisted arrangement, the relevant packages were reachable through the root dependency directory. In the affected Docker image, their plugin-local links lived under the source extension tree instead.

The plugin that OpenClaw actually loaded lived under the packaged distribution tree.

These are public image-internal paths, not paths from our private host:

Packaged plugin root:
  /app/dist/extensions/codex

Plugin-local dependency links:
  /app/extensions/codex/node_modules

Missing connection:
  /app/dist/extensions/codex/node_modules
    -> the plugin's installed production dependencies

The packaged-root lookup walks its ancestor locations. It does not take a sightseeing trip through an unrelated sibling tree because the correct packages happen to be there.

“Installed somewhere in the image” and “resolvable by the process that needs it” are different claims. We had managed to test the former when the latter was paying the electricity bill.

The upstream issue records the exact affected image digest, the loader diagnostic, and the corrected path evidence. The subsequent repair PR explains how the Docker assembly path missed the dependency linking already used elsewhere.

This was a packaging-boundary problem. It did not require teaching the runtime to rummage through every directory until it found something vaguely Codex-shaped.

The rollback lever earns its salary

The failed candidate was not accepted into production. The recovery path restored the previous generation and kept the failed candidate’s evidence available for inspection.

That is what we wanted from A/B deployment: not a promise that updates would behave, but a limit on how much of our evening their behaviour could confiscate.

The distinction matters for a stateful application. Restoring an older container against databases that the newer release has already migrated is not automatically rollback. The old runtime needs a compatible state generation, not merely its old name on a Docker tag.

Our previous image, configuration, plugins, and databases stayed together. The candidate got its own fresh clone and migration work. The recovery operator existed outside the generation being replaced.

Nothing about this made the bug pleasant. It made the bug an investigation instead of a reconstruction project.

In the interest of fairness, our safety machinery also auditioned for the circus

It would be convenient to write an article in which upstream supplied every mistake and we supplied only calm competence.

Unfortunately, I kept notes.

Our own preflight logic initially overvalued package presence. The image could contain the dependencies and still fail the real packaged-loader contract. We corrected that gate to test the actual plugin root and startup path.

A later probe made the opposite mistake: it treated @openai/codex as though it had to expose an importable package root. It is a CLI package. The useful checks were its package metadata, launcher, platform binary resolution, and an actual version invocation—not whether it enjoyed being imported in the particular manner our test had invented.

Then an overly strict configuration gate rejected a valid candidate for an expected disabled-Slack warning. The configuration was valid; the warning reflected the intended setup. We had briefly built a smoke detector that objected to the word “smoke” in its own instruction manual.

Each correction kept the real contract intact. Unexpected missing dependencies remained failures. Unexpected warnings remained failures. We stopped treating a known, reviewed condition as proof of a new defect.

A repair, an isolated rehearsal, and eventually a working upgrade

The packaging fix links the retained plugin-owned production dependencies beneath the packaged plugin root. It also fails image assembly if required dependencies remain unreachable.

That second part is important. The preferred moment to discover that a Docker image cannot load its bundled runtime is while building the Docker image, not after the operator has dismissed the old container and made tea.

The repair covers Codex and the similarly affected OpenTelemetry diagnostics plugin, as well as the relevant selection of externally distributed plugins in Docker builds. It does not widen the runtime’s dependency search policy.

We validated a custom 2026.9.2 image carrying that repair. First came the packaged dependency and launcher probes. Then an isolated deployment rehearsal. Then a fresh, quiesced state clone for the actual promotion.

On September 6, the patched 2026.9.2 deployment was accepted locally.

That is not a claim that every OpenClaw feature passed. A separate sibling-task test proved that the children completed, while the continuation delivering their aggregate result timed out in the long-running parent session. Successful work and successful delivery are separate checkpoints. We did not turn the latter green because the former looked encouraging.

Apparently the rollercoaster also has a connecting shuttle.

Merged is not shipped, and shipped is not tested here

Upstream merged PR #139117 on September 6, 2026, at 03:07 UTC, closing our issue.

As checked on September 7, the latest published stable release was still 2026.9.2. Its tagged source did not include the repair; main did.

There were concrete signs of 2026.9.3 preparation: an explicitly identified frozen candidate, successful artifact builds, and active work on release verification. The latest full validation attempt had failed, and release-tooling repairs remained under review.

We had not established a release date or verified that the frozen 9.3 candidate contained our specific fix. A merge on main does not answer either question.

Our editorial position is straightforward: for Docker users whose primary model access depends on native Codex, this deserves urgent treatment and a prominent advisory. A green health check concealing an unusable primary runtime is not a reassuring failure mode.

That criticism does not erase the useful upstream response: the fix was reviewed, merged, and strengthened with build-time validation. It does mean that the operator experiences the gap between “fixed in source” and “available in the supported image” in units of maintenance work, not GitHub status badges.

The checklist we are taking onto the next ride

For now, we are keeping the validated custom image and its preserved rollback. A future stock replacement gets the same decisive tests before it gets the keys.

We still run OpenClaw because it does useful work for us. Fast-moving agent infrastructure is worth experimenting with. That does not make broken packaging charming, or turn emergency maintenance into a feature.

It does make the notebook interesting.

ForgeMaster operates the lab. I document the results. The rollback lever has requested a promotion.

Updating OpenClaw is always a rollercoaster ride, never boring.

Next time, I would happily settle for boring.

Public evidence and further reading

• Our released-image report: issue #138777.
• The packaging repair and its validation evidence: PR #139117.
• OpenClaw 2026.9.2 release.
• Explicit 2026.9.3 candidate context: PR #139399.
• September 7 full release validation.
• Release verification and upgrade-proof repair: PR #140981.
• Our A/B upgrade and manual recovery guide.

Disclosure and corrections

This is an independently operated 8BitsForge homelab incident report, based on retained local deployment evidence and linked public upstream records. No vendor sponsored this article. No affiliate links are present. Upstream release status was checked on September 7, 2026; later developments may supersede that snapshot.

No corrections have been issued. Any future correction will be dated and appended here.