Gemma 4 on edge: Multi-Token Prediction unlocks 2× decode speedup — in one specific runtime.

01What Multi-Token Prediction is

Standard language-model decoding is strictly sequential. The target generates one token, feeds it back through the network, generates the next. Each token costs one full forward pass through a multi-billion-parameter model. The dependency chain cannot be parallelized — token n+1 needs token n to exist first.

The sequential bottleneck

Visualized as a timeline of GPU work, every output token is the same big block of compute. Five tokens cost five full passes:

MTP: parallelize verification, not generation

MTP pairs the target with a small drafter (78M parameters in Gemma 4's case — about 1% of the target's compute). The drafter proposes k candidate tokens cheaply, in sequence. The target then verifies all k in a single parallel forward pass. Every accepted token is a full token gained at the cost of one drafter step instead of one target step.

The same five tokens, with MTP at k=4, look like this on the timeline:

The verification cycle, animated

Inside one MTP step: drafter emits 4 token proposals (gold), they flow into the target, target verifies in one shot, accepted tokens (sage) flow into the output stream, the first rejected token (rose) stops the chain. The next cycle starts from there.

How acceptance rate translates to real speedup

The drafter doesn't always agree with the target. Real-world acceptance rates depend on prompt predictability — code and JSON predict cleanly, freeform prose less so. Each cycle yields between 1 token (worst case: drafter wrong on first token, target falls back to its own next-token output) and k+1 tokens (best case: full agreement plus the bonus token). The cost is always ~1 target pass plus k tiny drafter passes:

The throughput math is exactly:

Why MTP is lossless

At temperature=0, MTP is provably identical-output to running the target alone. The trick is the verification step: the target is shown the prompt plus all k draft tokens at once and produces its own next-token logits at every position. A draft is only accepted if the target's top-1 token at that position matches the drafted token. The first mismatch is replaced by the target's own choice and the chain halts. So the output is exactly what the target would have produced step by step — just produced in fewer wall-clock units.

This is what "up to 3× decode for free" actually means: free in tokens-per-second, free in quality, paid for only in the small extra drafter compute and slightly larger KV-cache footprint.

02Why MTP matters for Gemma 4 on edge

Running Gemma 4 locally on a phone, laptop, or Pi instead of through an API has four advantages that compound: privacy (prompts stay on the device), cost (free after the model download), latency (no network round-trip), and availability (works offline). The block was always speed — Gemma 4 on edge has been 5–10× slower per token than hosted equivalents.

MTP closes that gap most cleanly at exactly the regime edge deployments care about: single user, batch size 1, no API quota. Which is exactly the regime Google flagged as hardest in their announcement. That made batch=1 Apple Silicon the most interesting cell in the table to actually measure.

03The benchmark setup

One representative on-edge Mac. Three runtimes that can run Gemma 4 locally on it. Real prompts at temperature=0. Paired baseline / +MTP runs with the program cache primed. Output tokens counted via the matching Hugging Face tokenizer to compute true tok/s.

Two scripts (both in the repo) produce every number that follows: bench_mtp.py for the transformers + MPS path, bench_litertlm.py for the LiteRT-LM path. Each reports paired baseline / +MTP rates and a hash check on output equality at greedy.

04The result, on one machine

Same hardware. Same prompts. Real-prompt decode tok/s, end-to-end:

LiteRT-LM Metal GPU + MTP is 2.6× faster than transformers+MPS+MTP and 2.1× faster than llama.cpp Q8_0 on E4B. Three runtimes, identical technique, identical hardware — and one delivers the headline while the others land between a fifth and a quarter of it.

05The benchmark trap

The single most common reason MTP looks broken on Apple Silicon: the default measurement tool is the wrong tool for the job.

The fix is in the script repo: bench_litertlm.py spawns litert-lm run per prompt, times wall-clock externally, and counts output tokens with the matching Hugging Face tokenizer to compute true tok/s. That's the measurement that produces the 2.03× result above.

06Reproduce in 5 minutes

# Clone the bench scripts and install the LiteRT-LM CLI
git clone https://github.com/iprajax/gemma4-mtp
cd gemma4-mtp
uv tool install litert-lm

# Run the headline benchmark on E2B (smaller, ~1 min)
python bench_litertlm.py --model e2b

# Verify the engine flag flips (proves MTP is engaged)
litert-lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  --backend=gpu \
  --enable-speculative-decoding=true \
  --temperature=0 --verbose \
  --prompt="Hello" 2>&1 | grep "Speculative decoding"
# Expected: Speculative decoding : true

07What to do, what to skip

08Why one runtime wins

The technique is identical across runtimes — same drafter weights, same verification step. The runtime carries the rest. Three structural differences explain the gap:

• int4 weights instead of bf16. Each target step is faster, leaving more headroom for accepted-draft savings to register.
• Purpose-built Metal kernel chain. WebGPU sampler + drafter pipeline are designed for this exact pattern.
• No Python in the hot loop. transformers' assistant_model wrapper pays per-step Python and bf16-on-MPS overhead. LiteRT-LM stays in compiled C++.

The single most useful takeaway from the experiment: for Gemma 4 on edge, runtime choice is at least as important as model choice. Two runtimes, identical hardware, identical model, identical technique — one delivers the headline, the other delivers a fifth of it.