Drafters: draft models, Medusa, EAGLE, MTP, DFlash, DSpark
A language model writes one token at a time, and every token costs a full pass through the model. On a GPU that pass is limited by how fast the weights can be read from memory, not by arithmetic. Reading the weights once to score one position or to score five takes almost the same time.
Speculative decoding uses that spare arithmetic. A cheap drafter guesses the next few tokens. The large target model then scores all of the guesses in a single pass, keeps the ones it agrees with, and adds one token of its own. When the guesses are good, one expensive pass yields several tokens instead of one.
Two terms recur below. A model’s hidden state is the vector of numbers it computes for the current position just before it picks a token; it encodes what the model is about to say. The KV cache is the per-token memory a transformer keeps about everything already in the conversation, so that each new token does not re-read the whole text.
The drafter is the part that varies. This page explains the kinds in use, how they differ, and what to watch for. How many tokens to draft per step is a separate setting: see speculative tokens.
Why the answer does not change
The target checks every drafted token before it is kept:
- Greedy decoding. A drafted token is kept only if it is the token the target would have picked itself. At the first disagreement, the target’s own choice is used and the remaining guesses are thrown away.
- Sampling. Each drafted token is kept with probability min(1, p/q), where p is the target’s probability for it and q is the drafter’s. On a rejection, a replacement is drawn from the target’s distribution with the drafter’s probabilities subtracted. This rule makes the output statistically identical to sampling from the target alone (Leviathan et al. and Chen et al., 2023).
So the drafter only decides how fast text comes out. A bad drafter wastes work; it cannot change the text.
How each kind makes its guesses
| Kind | Reads | Passes for 4 tokens | Trained | Adds to memory |
|---|---|---|---|---|
| Draft model | tokens only | 4, in sequence | separately, as a small model | own weights, own KV cache |
| Medusa heads | last hidden state | 1, heads in parallel | added to a finished target | a few small heads |
| EAGLE | hidden state + latest token | 4, in sequence | added to a finished target | one small layer + its KV cache |
| MTP | same as EAGLE | 4, in sequence | with the target | one layer, inside the checkpoint |
| Block drafter | hidden states from several layers | 1, whole block | DFlash: added · DSpark: with the target | a small model or extra layers + KV cache |
Draft model. The oldest approach: run a much smaller model from the same family, for example a 1-billion-parameter model next to a 70-billion one. It needs no changes to the target, but it sees nothing of the target’s internal state. It guesses only from the text so far, so its guesses are the weakest per unit of size. It also keeps a complete KV cache of its own, which costs memory for every conversation being served.
Medusa. Several small heads sit on top of the target’s final layer. Head 1 predicts the next token, head 2 the one after, and so on, all from the same hidden state and all at once. That makes drafting very cheap. The catch is that head 3 guesses its token without knowing what heads 1 and 2 chose, so guesses further out are accepted less often.
EAGLE. One small transformer layer takes the target’s hidden state and the latest token, and predicts the next hidden state and token. It then feeds its own output back in to predict the one after. Because each guess builds on the previous one, acceptance holds up further out than with Medusa. EAGLE drafts a small tree of alternatives rather than a single line, so the target can check several candidate continuations in one pass; EAGLE-2 shapes that tree per context from the drafter’s own confidence, and EAGLE-3 feeds the layer with features from the target’s low, middle and top layers instead of the top one only.
MTP (multi-token prediction). The same shape as EAGLE, but trained together with the target as an extra training objective rather than added afterwards. DeepSeek-V3 popularised it, and many recent open models ship their MTP layer inside the checkpoint. Because it learned alongside that exact model, it matches it well. Inference engines often run it through their EAGLE code path.
Block drafters. These produce a whole block of guesses in a single pass instead of one guess per pass. DFlash (and its successor DFlash2) is a separate small model that reads hidden states from several of the target’s layers and fills in the block in parallel using block diffusion. DSpark is DeepSeek-V4’s built-in drafter. Like MTP it ships inside the checkpoint, but it drafts a fixed block of 5 tokens, so engines expect the draft depth to match that block size.
How they compare in practice
- Acceptance. Drafters that read the target’s hidden state (EAGLE, MTP, block drafters) guess better than a separate draft model of the same size, because they see what the target is about to say. A drafter trained with the exact model, like MTP or DSpark, usually guesses best of all.
- Drafting cost. Chained drafters (draft model, EAGLE, MTP) need one pass per guess, so drafting further ahead costs proportionally more. Medusa and block drafters need one pass for the whole block, but they always pay for the full block, even when only the first guess survives.
- Memory. Heads and single extra layers are small next to the target. A separate drafter is not: a 2-billion-parameter one is about 4 GB of weights at 16-bit precision before its cache. Drafters that keep their own KV cache also take memory away from the space that holds conversations, so fewer requests fit at once.
- Load. Speculation pays off most when the GPU serves few requests at a time, because that is when arithmetic sits idle. With many requests in flight, the GPU is already busy, verifying extra guesses is no longer free, and the speed-up shrinks or disappears.
Things that go wrong
- A drafter made for a different copy of the model. If its tensor shapes match, it loads and runs, but its guesses can be far worse. One DFlash2 drafter published with 93 % acceptance on its base model accepted only 14–28 % against a differently quantized export of that model (the cause was not isolated). The output stays correct; the speed-up is gone.
- A fine-tuned target with the original drafter. Output still follows the target, because the target checks every token. Acceptance drops wherever the fine-tune changed the model’s behaviour.
- A drafter stored in a different number format. Quantized checkpoints sometimes leave the drafter in another format than the main model. DeepSeek-V4’s NVFP4 release quantizes its experts as NVFP4 but leaves the DSpark drafter in MXFP4. An engine that assumes one format for every expert layer can fail to load the drafter. More on the two formats: MXFP4 vs NVFP4.
- Several tokens arriving at once. With speculation, one step can emit several tokens, for example the end of one tool call and the start of the next. Streaming parsers written for one token at a time can drop or merge characters at those boundaries. Test tool calling with speculation on.
Related settings: speculative tokens, KV type, memory budget.