Qwen3.8 Flash-Next on One Arc Pro B70

The model occupied 74.5 GB. The graphics card had 32 GB. This is roughly the computational equivalent of fitting a concert piano into a hatchback while still expecting someone to play it during the drive. Fortunately, local inference lets some of the instrument remain in system memory or on fast storage—provided the arrangement does not collapse into expensive furniture transport.

The first experiment was simply whether one Arc Pro B70 could run the model at a useful, repeatable speed. The second asked whether its clever acceleration options could improve that baseline without changing the answer. Success meant complete deterministic output and a material wall-clock gain; “the feature activated” was not invited to grade its own homework.

Experiment summary

The short version

Question

Can one 32 GB B70 run a 74.5 GB hybrid MoE quant at useful speed, and can exotic acceleration paths beat the simplest stable placement without changing output?

Control

Clean Vulkan placement at n-cpu-moe 16, 32K context, and Q8 KV.

Candidate

GPU expert caching, detached MTP, sidecar-free n-grams, and a newer llama.cpp master.

Outcome

The clean profile delivered about 15.15 wall tokens/s. Every acceleration arm lost speed, changed output, optimized a weaker placement, or measured replay instead of general work.

The question

Qwen3.8 Flash-Next is not a conventional 27B model wearing an ambitious name tag. The retained quantized artifact spans three GGUF shards and totals 74,538,755,776 bytes. It combines a mixture-of-experts core with a large PLE/n-gram lookup component and optional MTP speculation.

Nibbler has one 32 GB Intel Arc Pro B70 plus enough host RAM and fast local storage to make a split placement plausible. The practical question was:

Can one B70 run this model at useful single-request speed, and can any of its exotic acceleration paths beat the simplest stable placement without changing the output?

The answer was yes, followed by several increasingly educational versions of no.

Lab recipe

The target’s PLE/n-gram lookup tensor alone is about 27,465 MiB. It is SSD-backed through mmap and lazy row reads; it is not the same thing as speculative decoding, and it is not the detached MTP sidecar. This distinction becomes important once every feature begins using the word “prediction” and the lab assistant begins using the word “coffee.”

Prerequisites and provenance

• One Intel Arc Pro B70 with 32 GB dedicated VRAM, sufficient host RAM, and local SSD storage for lazy mmap reads.
• The three-shard Unsloth UD-IQ1_M target at revision c8b5954a88c2775c546b92593eda40ea041d3176, with a verified aggregate size of 74,538,755,776 bytes.
• llama.cpp commit d7bd3bfcad3e29c7e49fd26f38c79ee3e9a3fd6b, built with the Vulkan backend. The tested container image is exactly sha256:929f1c763fb5005f9078924dfb87d85a7e46fe012c76b9c13b342601e55debff and reports 0.3.0-dev built with GNU 16.1.1.
• Native logs proving Q8 K/V cache, one slot, Vulkan placement, and the approximately 27,465 MiB PLE tensor remaining mmap/lazy from SSD.

The evidence preserves the source commit and final image digest, but not the literal Docker build command. Rebuild from the pinned commit with Vulkan and verify the runtime identity, or use an image independently verified to the recorded digest. The argument block below is an equivalent reproduction recipe, not the literal historical invocation; model paths are sanitized.

--model /models/Qwen3.8-Flash-Next-UD-IQ1_M-00001-of-00003.gguf
--ctx-size 32768
--cache-type-k q8_0
--cache-type-v q8_0
--parallel 1
--n-cpu-moe 16
--threads 16
--batch-size 4096
--ubatch-size 1024
--flash-attn on

Vision and speculative decoding were off for the clean baseline. The historical harness also enforced mmap plus a CPU override/lazy-read rule for per_layer_token_embd.weight, but its literal flag spelling was not retained in the publication evidence. Do not fabricate it: verify the observed SSD-lazy tensor placement in native logs before accepting a run.

Controlled variables and evaluator

1. Verify the model revision/aggregate size, source commit, image digest, and runtime version before loading the model.
2. Start the n16 profile and require a clean concise-chat smoke.
3. Run one unmeasured 512-token warmup, then three complete nondegenerate 512-token deterministic generations. Record wall and native throughput plus output hashes.
4. Run the matched 1,031-token prefill twice and report first-pass and warm throughput separately.
5. For the placement edge, change only --n-cpu-moe from 16 to 15 in a fresh process. A startup execbuf ENOMEM or Vulkan DeviceLostError is an expected failure signal, not a reason to discard the n16 control.
6. For later acceleration arms, retain n16 unless the feature itself forces a matched placement change; always compare that arm both with its matched control and with the fastest clean n16 baseline.
7. Reject an arm on incomplete output, semantic/hash divergence under the deterministic policy, a native anomaly outside the declared resource-edge arm, or less than the predeclared material gain.

Expected baseline signals are repeated fresh n16 starts, complete clean output, approximately 15.15 wall tokens/s on the final confirmation, and roughly 302.8 prompt tokens/s on the warm 1K prefill. The expected n15 signal is startup failure at the Vulkan resource boundary. If n15 starts reliably on a later runtime, that is a new experiment, not permission to borrow this verdict.

Finding the clean placement

The principal placement control was n-cpu-moe: how much MoE work remained on the CPU side so the rest could fit and execute on the B70.

The best repeatably stable clean configuration was:

n-cpu-moe = 16
threads = 16
batch = 4096
ubatch = 1024
context = 32768
KV = Q8
backend = Vulkan

The confirmation result measured approximately 15.15 wall tokens/s. Later matched mixed-workload controls generally landed around 14.6–14.7 wall tokens/s.

The notebook now had a baseline, an edge, and enough arrows in the margin to qualify as a small transit map.

One step more aggressive—n-cpu-moe=15—crossed the practical memory boundary. Startup failed with Intel execbuf ENOMEM followed by Vulkan DeviceLostError. The n16 placement survived repeated fresh starts; n15 did not.

This gave us a useful baseline and a clean rule: the fastest profile that starts once is a candidate; the fastest profile that survives repeated fresh starts is a configuration.

Acceleration attempt one: GPU expert cache

An unmerged expert-cache branch offered an appealing idea: keep a working set of experts on the GPU and avoid some movement.

The best tested cache arm used 48 slots and approximately 2,521.8 MiB. It was:

• 2.03% slower than the matched patched-off control; and
• 28.48% slower than the tuned clean n16 baseline.

Worse, cache-enabled fixed-seed runs produced divergent hashes while the clean and patched-off controls were exact.

The cache was therefore slower than the profile it was meant to improve and less deterministic than the branch with the feature disabled. It had achieved the rare optimization hat trick of consuming memory, losing speed, and creating a new argument about correctness.

Verdict: Rejected on this runtime and hardware. Larger caches were not pursued.

Acceleration attempt two: detached MTP

The optional detached MTP sidecar was Q8_0, 4,142,897,248 bytes, revision 67de7592b670ef454a903574d5e2aa6c8e1d6b46.

The first lesson was operational: --spec-draft-ngl 0 alone does not prove a CPU-only draft model. A genuinely device-free sidecar required both:

--spec-draft-device none
--spec-draft-ngl 0

Native logs then had to confirm zero draft Vulkan buffers. The verified CPU placement used approximately 3,940.54 MiB for sidecar weights, 64 MiB for KV, and 256.04 MiB for compute in host memory.

The performance result was decisive:

The GPU-resident sidecar first fit at the slower target placement n24. It improved that compromised control, but barely exceeded the best simple n16 profile—and deterministic output diverged.

Verdict: Closed. The sidecar won against the parking spot it forced us to use, not against the best car already on the track.

Acceleration attempt three: sidecar-free n-grams

The runtime also offered n-gram speculative strategies without a detached model.

Then ngram-mod appeared to improve throughput by roughly 34%.

This was exciting for exactly as long as it took to notice that a shared hash pool was reusing identical prior outputs. The headline measured a replay-cache ceiling, not general task acceleration.

We reran the comparison in fresh processes and separated three cohorts:

• First exposure: inconclusive because control drift reached 14.61%.
• Novel requests: controls were stable; candidate gain was only 1.99%, and ngram-mod drafted zero tokens.
• Exact replay: controls were stable; ngram-mod was 16.11% slower, accepted 160/500 drafts, and changed all six outputs.

Verdict: Rejected as a general acceleration path. Stateful caches must report first exposure, novel input, and exact replay separately. Mixing them produces a splendid number and a useless conclusion.

Acceleration attempt four: newer llama.cpp master

Upstream commit c841aeeb8bb2fe417038dadfa9b007cf1a9ef950 included a merged Vulkan MoE padding change that might have reduced the allocation pressure behind the n15 failure.

The matched result was technically clean but operationally unconvincing:

At n16, c841 averaged 14.3701 wall tokens/s versus 14.2069 for d7bd. It was exact and deterministic, but the workloads disagreed too sharply for the small aggregate gain to justify promotion.

At n15, the newer build still failed at startup with the expected resource-boundary classification. The practical placement edge had not moved.

Verdict: Retain d7bd/n16. Revisit only for a materially different upstream change or a dedicated long-context benefit test.

What we learned

Operational verdict

Retain the artifacts, d7bd runtime, n16 configuration, evidence, and restart-ready handoff. Do not add Flash-Next to the live Model Router or OpenClaw catalog from this experiment alone.

Every experiment restored production: the Router returned healthy and non-faulted with the production B70 route active, zero inflight requests, no canary, unchanged OpenClaw configuration, and unrelated container identities preserved.

The result is still valuable. One consumer B70-class card can run a model whose quantized artifact is more than twice its VRAM at a usable batch-worker rate. We merely failed to discover the magical checkbox that vendors traditionally keep next to the unlimited memory switch.

Disclosure and corrections

This experiment used 8BitsForge-owned equipment. No vendor sponsored the test, supplied hardware, reviewed the conclusion, or paid for the result. No affiliate links or third-party advertising are enabled.

No corrections have been issued. Any future correction will be dated and appended here.