JOURNAL / MODELS & COMPUTATION

What is speculative decoding?
Draft fast. Verify together.

Language models usually generate one token at a time. Speculative decoding asks a faster draft process to suggest a short continuation, then lets the target model check those candidates together. When enough candidates are accepted, the response can advance with fewer serial target-model steps.

Black and white chess pieces in a row, with the center king in focus and the surrounding pieces softly blurred.DECODING / CANDIDATES / SYMPLOKA
PROPOSE A PATH. CHECK IT.

A good guess saves a turn through the model.

Speculative decoding lets a lightweight drafter sketch several possible next moves. The target model verifies the candidate sequence before the accepted tokens become part of the answer.

01 / WHY GENERATION IS SERIAL

The next token depends on the one just chosen.

In ordinary autoregressive decoding, the target model processes the current context, selects one token, adds it to the sequence, and runs again. Each step depends on the previous result. Even when a GPU can do substantial parallel work inside one model pass, the output still advances through a chain of dependent steps.

This can leave compute hardware waiting on memory movement and sequential decisions. Speculative decoding reduces some of those serial target-model calls by checking a short block of proposed tokens together. It targets the generation phase after prompt processing; it does not shorten the prompt or make the model’s parameters smaller.

For the full path from prompt prefill to output tokens, start with the inference guide ↗.

02 / DRAFT, THEN VERIFY

The target model still decides what the answer says.

  1. Draft a short continuation. A smaller model, lookup method, or added prediction heads propose candidate next tokens.
  2. Score candidates together. The target model evaluates the proposed continuation in a parallel pass using the same prefix.
  3. Keep the accepted prefix. If the candidates match what the target model would generate, several can be committed from one target pass.
  4. Correct the first mismatch. When a candidate fails verification, the decoding method produces a valid target-model continuation from that point and discards later draft guesses.

In canonical speculative sampling, rejection sampling and a correction step preserve the target model’s output distribution. That means the probability distribution stays the same; it does not mean separate random runs produce the same sentence.

For greedy decoding, a candidate is accepted only while it matches the target model’s next-token choice. Sampling needs the acceptance-and-correction rule described in the original speculative decoding paper ↗ and speculative sampling study ↗. Other implementations may make different trade-offs, so “speculative” alone does not promise unchanged behavior.

03 / MORE THAN ONE KIND OF DRAFTER

The candidate generator can be a model, a lookup, or an added head.

Smaller draft modelA faster model proposes tokens for a larger target model to verify. This is the classic arrangement; keeping a second model loaded adds memory and compute overhead.
Prompt or n-gram lookupA system retrieves likely continuations from repeated patterns in the prompt or previous text. It can avoid a separate learned draft model, though it depends on useful repetition.
Multiple prediction headsMethods such as Medusa add heads that propose tokens at different future positions. A tree of candidates can then be verified together; training or adapting those heads is part of the design.
Feature-level draftingEAGLE predicts internal model features to form candidate continuations. This changes the draft mechanism while retaining a verification step against the target model.

These approaches share a draft-and-verify pattern, but they differ in extra memory, training, draft cost, and how candidates are organized. The Medusa paper ↗ explores multiple decoding heads and tree-based verification; EAGLE ↗ drafts from intermediate features.

Draft and target runtimes may both need request state. See how KV cache uses memory during generation ↗.

04 / WHEN IT HELPS

Speed depends on acceptance and verification cost.

Drafts are cheapThe candidate generator must produce tokens quickly enough that its added work is offset by fewer target-model steps.
Many tokens are acceptedLonger accepted prefixes let one target pass advance the response farther. Low acceptance means more discarded draft work.
Verification is efficientThe target model has to check the candidate block. If verification dominates runtime, theoretical gains shrink.
The workload fitsOutput length, request batch size, memory pressure, model pair, and decoding settings can all change the result.

A 2026 study of several speculative methods on vLLM ↗ reports that verification by the target model dominates execution in its tested setups, while acceptance length varies across positions, requests, and datasets. The study also finds a gap between theoretical speedup bounds and observed performance. Those results reinforce a practical point: do not copy a paper’s multiplier into a different serving workload.

Speculative decoding does not guarantee lower time to the first token, lower total cost, or higher throughput. It is an optimization to measure against ordinary decoding on the same hardware, model, request mix, and concurrency.

05 / HOW TO TEST A CLAIM

Compare complete requests under representative load.

  1. Choose a realistic request set. Include prompt lengths, answer lengths, task types, and sampling settings from the intended use.
  2. Record both latency phases. Track time to first token separately from time per output token and full response time.
  3. Measure acceptance and overhead. Record draft cost, accepted tokens per verification pass, target verification time, and extra memory.
  4. Test the actual batch size. Measure at the concurrency and queueing conditions expected in operation, not only with one isolated request.
  5. Check output behavior. Confirm the exact decoding method and compare task quality or distribution guarantees for the implementation being evaluated.

For model and runtime trade-offs, continue with a model-selection checklist ↗ or a whole-system cost framework ↗.

PRIMARY PAPERS & SYSTEM EVALUATIONLeviathan, Kalman & Matias, Fast Inference from Transformers via Speculative Decoding (ICML 2023) ↗Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling ↗Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads ↗Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty ↗Liu et al., Speculative Decoding: Performance or Illusion? (2026 arXiv preprint) ↗