Squeezing Qwen3.8 Until It Cried: A 46.9% Speedup and One Optimization Too Far

Picture asking a warehouse worker to drag the same enormous filing cabinet across the room several times for every word you type. Our Qwen3.8 setup had the silicon equivalent: a few large parts of the model were being read repeatedly while it drafted and checked each token. Most of the model was already compressed; these holdouts were still taking the scenic route through memory.

The experiment was simple: shrink those repeatedly read parts before the server started, then measure whether the GPU could generate text faster. The bargain had teeth. Speed counted only if the model still reasoned, used tools, understood images, handled long context, survived a restart, and could be rolled back cleanly. Faster nonsense remains nonsense, merely delivered with admirable punctuality.

We call the resulting checkpoint baked. In this article, that means quantizing two expensive parts ahead of time—the shared output head that scores possible next tokens and the multi-token prediction (MTP) draft tensors—then storing them directly in the model files as ready-to-load GPTQ INT4. Instead of leaving those parts as BF16 or FP16 tensors for the runtime to load and use, the smaller form is already “baked into” the checkpoint. It is convenient project shorthand, not a special feature of the Qwen architecture; from here onward, we will use it freely.

Experiment summary

The short version

Question

Can moving Qwen3.8's shared output head and MTP draft tensors from BF16/FP16 into native on-disk GPTQ INT4 make one Arc Pro B70 materially faster without giving away model behavior?

Control

The existing SergiioB Qwen3.8-27B GPTQ-Int4 body on the pinned vLLM XPU 0.27 execution contract.

Candidate

Launch80's baked-v1 checkpoint, which keeps the model body byte-identical while storing the shared lm_head and MTP draft tensors as GPTQ INT4 G128.

Outcome

Matched decode rose from 65.36 to 96.01 tok/s (+46.88%). The candidate passed the broader quality, tool, vision, long-context, restart, and recovery qualification and entered production.

On our Intel Arc Pro B70, that change moved matched Qwen3.8-27B decode throughput from 65.36 to 96.01 tokens per second. The result survived a much broader qualification and became the production route. Then a second, apparently obvious optimization won a quick screen by 14.7%, lost the comprehensive comparison, and was politely escorted away from the Model Router.

This is the story of both results, because publishing only the victorious half would teach exactly the wrong lesson.

The question

Qwen3.8-27B uses multi-token prediction (MTP) for speculative decoding. The draft proposes several tokens; the full target verifies them. On our MTP4 configuration, one generation step can read the vocabulary projection repeatedly: four draft passes plus the target verification pass.

The model body was already GPTQ INT4. The expensive loose ends were the shared lm_head vocabulary table and the MTP draft tensors. Launch80’s baked checkpoint stores those components on disk in the same symmetric GPTQ INT4 G128 format as the body:

The important word is shared. Quantizing a detached draft model can change acceptance while leaving target logits untouched. Quantizing a head used by the target changes the target itself. A speed screen could tell us whether the bytes mattered; it could not tell us whether the resulting model still deserved production traffic.

The hypothesis therefore had two parts:

1. The baked head should materially reduce memory traffic and increase single-stream decode throughput.
2. The resulting target must still pass the behavioral, quality, tool, vision, context, restart, and recovery gates already expected from production.

Lab recipe

The matched screen started the control and candidate once each, with zero retries. Both used the same image, runtime overlays, GPU, context, FP8 KV cache, prefix caching, MTP depth, prompt, seed, and generation length. The intentional differences were the checkpoint and the baked-head gate required to load its native GPTQ tensors.

The portable serving shape was:

vllm serve Launch80/Qwen3.8-27B-GPTQ-Int4-baked \
  --quantization gptq \
  --dtype float16 \
  --max-model-len 209664 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

That command is a readable configuration summary, not a claim that stock vLLM reproduces our exact image. The tested runtime used a pinned Intel XPU build and pinned correctness overlays. Anyone reproducing this work should follow the current guidance in Launch80/B65 and the linked Intel Arc Pro B70 cookbook, then verify the actual loaded package versions, patches, device identity, head format, and MTP counters before trusting a number.

The matched result

Every measured request reached its fixed 512-token length, and each arm was internally stable. Cross-arm output hashes differed, which was expected: the candidate changed a target-visible head. We treated that as a reason to qualify the model, not as permission to wave a throughput table until everyone became too tired to object.

MTP acceptance fell in this small screen from 68.57% to 62.16%, yet decode still improved sharply. The reduction in bytes read outweighed the lost acceptance. This is a useful reminder that acceptance is an input to throughput, not throughput itself.

The part where the speedup had to earn production

The candidate then ran through two broader stages.

Stage One exercised behavior hierarchy and tool restraint, structured reasoning, quality, vision, long context, simple tools, open-ended responses, streamed tool chains, MTP activity, and fatal-runtime checks.

Those quality scores matched the frozen production-control floors. This was not a claim that every benchmark improved; it was evidence that the faster checkpoint stayed inside the accepted production envelope.

Stage Two tested the plumbing that benchmark posts often omit: OpenClaw text and native-image transport, novel versus exact-replay prefix-cache state, exact marker recall at a 179,500-token prompt, and one deliberate same-container restart. The cache replay reported 18,304 prefix hits with identical visible output. After restart, the container returned healthy in 181 seconds, the exact smoke test passed, MTP resumed, OOMKilled remained false, and the fatal-anomaly evidence stayed empty.

Only then did we add the checkpoint to the Model Router. The promotion transaction retained a tested rollback path, restored the baked vLLM 0.27 profile as the active route after verification, preserved the authenticated boundary, and changed no unrelated defaults, fallbacks, schedules, or weight stores.

The tempting 64K sequel

Once the baked checkpoint was working, another optimization looked obvious. The draft proposes tokens; the target verifies them. Why should the draft score all 248,320 output rows if the workload mostly uses a much smaller subset?

We built a workload-ranked 65,536-row draft vocabulary on the baked vLLM 0.29 profile. The first small screen was delightful: +14.709% median decode versus the full-vocabulary control.

Then we ran the comprehensive counterbalanced comparison.

That run covered 42 matched performance samples and 39,424 generated tokens per arm across two fresh-start epochs. The 64K head was operationally viable and correctly used full-vocabulary fallback for rare out-of-set tokens, but it saved only about 40 MiB of non-Torch weight memory and added just 1,452 reported KV tokens. The larger speed claim did not survive a broader workload.

The comprehensive run also found a shared model weakness: both arms often obeyed a later user instruction over a persistent system marker. Because the full-vocabulary control failed the same gate, the formal all-purpose qualification verdict remained inconclusive. That does not turn the independent throughput channel into a win. It means exactly what it says: neither vocabulary fixed the behavior, while the 64K candidate failed its predeclared performance target.

One harness defect deserves mention. The fixed-length speed prompts deliberately consumed their 256/512/2048-token budgets in the reasoning stream and ended with finish_reason=length, but an overly strict synthesizer also required visible prose. It refused to score the otherwise matched throughput cells. We recovered the comparison directly from the retained raw samples and kept the semantic and performance conclusions separate. The lab assistant has since written “a benchmark harness is also a candidate” on the whiteboard in letters large enough to be considered infrastructure.

What we learned

Operational verdict

The production route remains the baked-v1 full-vocabulary profile on vLLM XPU 0.27. It carries the large, qualified speedup. The ranked-64K candidate is retired.

That is a less exciting ending than stacking 46.9% and 14.7% into a single heroic number. It is also the ending that survives contact with the evidence.

Disclosure and corrections

This experiment used 8BitsForge-owned equipment. Launch80, Qwen, SergiioB, Intel, and vLLM did not sponsor the test, supply hardware, review the conclusion, or pay for the result. No affiliate links or third-party advertising are enabled.

No corrections have been issued. Any future correction will be dated and appended here.

Sources and corrections

• Launch80/Qwen3.8-27B-GPTQ-Int4-baked
• Launch80/B65, including the bake tooling, quality gates, serving notes, and raw benchmark records
• 8BitsForge retained matched-screen, qualification, promotion, and 64K comparison evidence; all cited campaign archives passed their recorded integrity manifests

Corrections will be appended here with a date and explanation.