How ubatch=1024 Won the Benchmark and Lost the Ability to Speak

GPUs process work in batches. A larger batch can be like carrying more groceries per trip: fewer journeys, better use of the hardware, and a pleasing number on the receipt. It can also be like choosing a basket too wide for the doorway. The trip count improves magnificently because nothing useful leaves the kitchen.

We tested one deliberately small change: double the model’s internal micro-batch from 512 to 1024 and see whether long prompts loaded faster. The win counted only if ordinary conversation, answer quality, speculative decoding, and structured tool calls still worked. The prompt benchmark improved by six percent. The model then failed five conversations and greeted the weather tool with 256 copies of <unused32>. Technically, it had found a way to communicate our verdict early.

Experiment summary

The short version

Question

Can a larger Vulkan micro-batch improve long-prompt ingestion without harming generation or tools?

Control

ubatch 512 with Gemma 4 12B QAT, native MTP depth two, Flash Attention, and Vulkan.

Candidate

The same complete profile with only ubatch raised to 1024.

Outcome

Prefill improved 6.0%, then chat throughput collapsed, correctness fell to 0/5, and tool calling emitted 256 copies of <unused32>.

The question

We were evaluating Gemma 4 12B QAT as a possible local agent model on Nibbler’s Intel Arc integrated GPU. The known-good configuration was already interesting:

• Gemma 4 12B QAT in UD-Q4_K_XL.
• A matching Gemma MTP draft model at draft depth two.
• llama.cpp build b9993 at commit 2969d6d15.
• Vulkan with every target and draft layer offloaded.
• Flash Attention enabled.
• Batch size 2048 and the default micro-batch size 512.

Would raising --ubatch-size from 512 to 1024 improve longer prompt ingestion without changing generation quality or tool behavior?

It was a good hypothesis. This is important, because the machine is about to make it look foolish.

Lab recipe

Prerequisites and provenance

• An Intel Arrow Lake-P Arc Pro 130T/140T integrated GPU using Vulkan on i915 and Fedora kernel 7.1.3-200.fc44.x86_64.
• llama.cpp build b9993 at commit 2969d6d15d67a08e7b83f26164b15350c79c5248 with the Vulkan backend enabled.
• The target and draft GGUF files listed above, verified by both exact byte size and SHA-256 before startup.
• Enough device and host memory to load every target and draft layer on Vulkan, plus native startup logs proving that placement and flash_attn = enabled for both models.

The historical record retains the runtime commit and verified artifacts, but not the literal Docker build invocation or the complete original container command. Accordingly, the following is an equivalent reproduction argument fragment, not a claim that this exact shell text was used. Paths are deliberately generic; the flags are the recorded experiment contract.

--model /models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
--spec-draft-model /models/mtp-gemma-4-12B-it.gguf
--spec-type draft-mtp
--spec-draft-n-max 2
--batch-size 2048
--ubatch-size 512
--flash-attn on

For the candidate, change one line and nothing else:

--ubatch-size 1024

The evidence confirms full target/draft Vulkan offload, but does not retain the literal device-selection and layer-offload flags from that service. Do not guess them. Require the native log to prove all layers are on Vulkan, MTP initialized at depth two, and Flash Attention is enabled before collecting a result.

Controlled variable and evaluator

1. Start the micro-batch 512 control and require clean startup plus a concise chat smoke.
2. Run the same 2,335-token prefill prompt three times; record every prompt-throughput value and keep cold and warm results separate.
3. Start the micro-batch 1024 candidate under otherwise identical conditions and repeat the same three prefill runs.
4. Regardless of the prefill result, run five chat generations capped at 512 tokens, three fixed 512-token raw generations, and one structured weather-tool request.
5. Record finish reason, output correctness, raw/chat throughput, tool name and arguments, drafted-token count, accepted-token count, and native anomalies.
6. Restore micro-batch 512 and require the concise chat and structured tool-call smoke to pass again.

Expected control signals are normal stops, five correct chats, a clean get_weather call for Paris, and nonzero MTP acceptance. The historical candidate failure signals are five finish_reason: length chats, no valid tool call, repeated <unused32>, and 0/507 accepted draft tokens. A prefill gain does not overrule any of those failures.

The known-good control

Flash Attention had already transformed prompt processing from 2.565 prompt tokens/s to 78.660 prompt tokens/s on an 82-token tool request and reduced target-model graph splits from 18 to 2. That gave us a credible control, not a permission slip to stop checking.

Before touching the micro-batch, the complete MTP-plus-Flash-Attention profile had passed:

• Five chat generations averaged 16.7467 tokens/s.
• The first cold chat run measured 14.3893 tokens/s.
• Four warm chat runs averaged approximately 17.3361 tokens/s.
• Three fixed 512-token raw generations averaged 19.0127 tokens/s.
• The structured weather request produced a clean get_weather({"city":"Paris"}) call.
• Tool generation measured 19.276 tokens/s.
• All ten MTP-drafted tool-call tokens were accepted.
• Every output stopped normally and passed its correctness check.

This was not merely a fast-looking server. It could still converse, stop, and use a tool—the three hobbies we generally expect from an agent model.

The tempting benchmark

The long-prefill A/B used the same 2,335-token prompt three times per configuration.

The candidate improved the overall average by 6.0% and the warm average by 7.4%. Average prompt latency fell by roughly 5.5%. It cleared the predeclared keep threshold of 58.5 prompt tokens/s.

At this point the lab notebook said the setting should remain in the experimental profile.

That sentence aged for approximately eight minutes.

The complete evaluator arrives carrying a chair

The next gate repeated the standard chat, raw-generation, and tool-call checks. This was mandatory: the micro-batch result could not be accepted from prefill alone.

The five chat runs all failed.

The candidate did not merely lose its 6% gain somewhere else. Generation throughput collapsed by roughly 60%, every chat ran into the maximum-token wall, and the outputs failed validation.

Then came the tool request.

Instead of returning get_weather({"city":"Paris"}), the model emitted <unused32> repeatedly for all 256 generated tokens. It reached finish_reason: length, returned no tool call, drafted 507 speculative tokens, and accepted zero of them.

The raw harness happened to report output_ok: true, but a broad success bit cannot overrule five broken conversations and a weather tool call replaced by 256 copies of the same unused token. The weather remained unknown. The laboratory atmosphere was crystal clear.

What we observed—and what we did not prove

The benchmark did not lie

The prefill benchmark correctly measured faster prefill. Our first question was simply too small.

For an agent workload, the acceptance question is not:

Did one phase become faster?

It is:

Did the complete workload remain correct, recoverable, and faster enough to matter?

A tuning change can improve prompt ingestion while corrupting state used later by decode. It can preserve raw text while breaking tool syntax. It can look wonderful for three short runs and then fail the first behavior the system exists to perform.

This is why our model-admission and tuning gates now require representative chat, raw generation, tool selection, tool continuation, long-context behavior where relevant, and post-failure recovery—not just a tokens-per-second screenshot wearing its best jacket.

Operational verdict

We rejected --ubatch-size 1024 for Gemma 4 12B QAT with MTP depth two, Flash Attention, Vulkan, and llama.cpp b9993 on this Intel GPU.

The only validated Gemma profile remained:

Gemma 4 12B QAT UD-Q4_K_XL
+ native MTP draft depth 2
+ Vulkan Flash Attention
+ batch 2048
+ micro-batch 512

ForgeMaster ultimately stopped the Gemma tuning effort and retained Ornith for the live agent workload. The experimental Gemma service was retired conservatively while its model files and negative evaluator evidence were preserved; the live Ornith service remained outside the cleanup scope.

That is not a failed experiment. We paid a few benchmark runs to avoid promoting a configuration that was faster only until someone asked it a question.

Do not reuse these throughput values as a general Gemma 4 or Intel GPU benchmark. They describe one historical model/runtime/hardware configuration and one carefully isolated failure.

Disclosure and corrections

This experiment used equipment owned by 8BitsForge. No vendor sponsored the test, supplied the hardware, reviewed the conclusions, or paid for the result. No affiliate links or third-party advertising are enabled.

No corrections have been issued. Any future correction will be dated and appended here.