Speculative Decoding: Faster On-Device LLMs

Running a language model on a device is usually slower than people expect, and not because the processor is short of arithmetic power. On a phone or single-board computer, generating text one token at a time is limited by how fast the model weights move from memory into the compute units. Speculative decoding is a way to raise that ceiling: it produces the same output as ordinary decoding but pays the memory cost less often.

Why on-device decoding is slow

An autoregressive model generates one token per forward pass. At batch size one, producing that single token means reading every weight in the model from memory, multiplying it by a small activation vector, and discarding it. The arithmetic per weight is tiny, so the processor spends most of its time waiting for data. In roofline terms, single-token decoding is memory-bandwidth bound, not compute bound.

The numbers make this concrete. A Jetson Orin Nano 8GB reaches about 68 GB/s over its LPDDR5 bus. A 7-billion-parameter model stored at 4 bits per weight is roughly 4 GB. Reading 4 GB at 68 GB/s takes about 59 milliseconds, so the memory system alone caps decoding at under 20 tokens per second before a single useful multiply is counted. Faster arithmetic units do not move that ceiling, because the ceiling is set by data movement. This is the constraint the technique is designed to attack.

The key idea behind speculative decoding

Much of the text a large model produces is easy to predict. Function words, common phrasing, and the obvious continuation of a sentence do not need the full model to be guessed correctly, so a much smaller model can propose them quickly. The large model is then used only to verify the proposals, not to generate each token from scratch.

Verification is the important part. Speculative decoding is exact, not approximate. The large model runs once over the whole batch of proposals, and a carefully chosen acceptance rule keeps only the prefix that is consistent with what the large model itself would have sampled. The rule is built so that the final output has the same probability distribution as ordinary decoding from the large model. The text you get is statistically identical to running the large model alone; only the speed changes.

How it works, step by step

One decoding step looks like this:

1. The small draft model generates a short run of candidate tokens, for example eight. This is cheap because the model is small.
2. The large target model runs a single forward pass over all eight positions at once, so its weights are read from memory only once for the whole group.
3. The candidates are compared against what the target model would accept, and tokens are kept up to the first point where draft and target disagree.
4. At the disagreement point the target model’s own prediction is used, so at least one correct token is always produced per step, and the process repeats.

If the draft is good, several tokens are accepted per step, and the device produces multiple tokens for the price of one target-model memory read. If it is often wrong, fewer are accepted and the benefit shrinks toward ordinary decoding.

Why the accepted tokens are almost free

The saving comes back to the memory-bandwidth wall. Reading the target model’s weights is the expensive step, and that read happens once per forward pass regardless of how many token positions the pass covers. Processing eight positions instead of one turns a set of tiny vector multiplies into slightly larger matrix multiplies, and on memory-bound hardware there is spare arithmetic capacity to absorb that extra work. So the additional positions cost almost nothing beyond the memory read you were already paying for. Every proposed token that survives verification is one you did not fetch the weights again to produce.

What the numbers look like

The original demonstration reported a 2x to 3x wall-clock speed-up on a large encoder-decoder model, with output identical to the standard implementation. The exact figure on your device depends above all on the acceptance rate: the fraction of drafted tokens the target model keeps. A high acceptance rate on predictable text yields larger speed-ups; a low rate on hard, high-entropy text yields little.

Production inference engines now implement this directly. In llama.cpp, you enable it by passing a small draft model alongside the target model:

raghu@techveda.org:~$ ./llama-cli -m qwen2.5-7b-q4.gguf -md qwen2.5-0.5b-q4.gguf -p "Explain what a spinlock is"

Here -m is the target and -md (also --model-draft) is the draft model. The number of tokens proposed per step is tunable with a separate draft-length flag. No retraining or architecture change is required; the draft only needs to share the target’s vocabulary so their tokens line up.

Where it breaks down

It is not a universal win. Several conditions can cancel the benefit:

• Low acceptance rate. If the draft model rarely agrees with the target, the extra draft and verification work can make generation slower than plain decoding.
• Memory pressure. The draft model must stay resident in DRAM alongside the target model and its key-value cache, which may be exactly the footprint a constrained board cannot spare.
• Shared-memory contention. On a unified-memory SoC the draft and target share the same processors and LPDDR bus, so a draft that is too large eats into the bandwidth you were trying to save.
• Compute-bound regimes. If you are already serving a large batch, or running a small model on a fast accelerator, the target pass is compute bound rather than memory bound, and the “free” verification is no longer free.
• Vocabulary mismatch. The draft and target must use compatible tokenizers, limiting which model pairs you can combine.

What this means for embedded and kernel engineers

The lesson generalizes beyond language models. When you profile an on-device inference workload and find the accelerator idle while memory traffic is saturated, the bottleneck is data movement, and the fix is to move less data or to reuse each transfer for more work. This method reuses a single weight read to settle several tokens; the same reasoning guides weight quantization, operator fusion, and how you place tensors across a shared memory hierarchy. On single-batch edge inference, the common case, memory bandwidth is usually the number to design around. These foundations — where the bytes move, and how the memory subsystem and scheduler behave under that load — are comprehensively covered in our training programs.

Key takeaways

• Single-stream, on-device decoding is limited by memory bandwidth, not by arithmetic throughput.
• Speculative decoding uses a small draft model to propose tokens and the large model to verify them in one pass, producing identical output to ordinary decoding.
• Accepted tokens are nearly free because the target model’s weights are read from memory once per verification pass.
• Reported speed-ups are around 2x to 3x, but the real figure depends on the draft model’s acceptance rate.
• The technique can lose its advantage under memory pressure, low acceptance, shared-bus contention, or compute-bound serving.

Frequently asked questions

Does speculative decoding change the model’s output?
No. The technique is designed to be lossless. The target model verifies every proposed token with an acceptance rule that preserves its original output distribution, so the generated text is statistically identical to standard decoding from the target model.

Why does speculative decoding help most at batch size one?
Single-stream decoding is memory-bandwidth bound: the processor reads the whole weight set to produce one token and spends most of its time waiting on memory. Verifying several proposed tokens in one pass reuses that single memory read, so the extra work is nearly free. At large batch sizes the target pass is already compute bound and the gain shrinks.

What does the draft model cost on a memory-constrained device?
The draft model must also stay resident in DRAM along with the target model and the key-value cache, and on a shared-memory SoC it competes for the same bandwidth. If it is too large, or its predictions are rejected often, it can erase the benefit.

How do I try speculative decoding without changing my model?
Tools such as llama.cpp support it directly. You pass a small draft model with the -md or –model-draft option alongside the target model. No retraining or architecture change is needed, and the draft only needs a compatible vocabulary.

Further reading

• Yaniv Leviathan, Matan Kalman, Yossi Matias, “Fast Inference from Transformers via Speculative Decoding”, ICML 2023 (Oral). arXiv:2211.17192
• llama.cpp speculative decoding documentation. github.com/ggml-org/llama.cpp
• NVIDIA Jetson Orin Nano series datasheet (memory bandwidth figures). Jetson Orin Nano datasheet (PDF)