Speculative tokens (num_speculative_tokens)

Result as of 2026-10-24. Every speculative profile on the 2x DGX Spark (GB10) pair runs at depth 4. Depth 6 was slower on real prompts, and 7, 8 and 16 were slower at 8 concurrent streams. Deeper drafts only won with a single stream. DeepSeek-V4-Flash runs at depth 0 because its drafter does not load yet, not because 0 was tuned (see Drafters).

What the setting does

• A small drafter proposes k tokens. The target model scores all k plus one more position in a single forward pass.
• The target keeps the drafted tokens up to the first one it disagrees with, then adds its own token at that spot. If all k match, it adds the token after them. So every step yields between 1 and k + 1 tokens.
• Depth 0 turns speculation off. The panel accepts 0 to 8.
• Decode on this pair is memory-bandwidth bound: each pass reads the weights once whether it scores 1 position or 5. Accepted drafts are tokens you get without paying for another weight read.

Expected tokens per step

If each drafted token is accepted with the same probability a, independently of the others:

E[tokens per step] = 1 + a + a² + … + a^k = (1 − a^(k+1)) / (1 − a)

The assumption is that acceptance is the same at every draft position and that positions are independent. Real drafters fail together on hard stretches of text and are rarely exactly geometric. The closest check here is DFlash2 at depth 16 (2026-10-02). There, the share of drafts that survived to positions 0–4 was 78.0, 60.1, 44.3, 33.7 and 25.7 %. Each position keeps 0.73–0.78 of the one before it, which is near-geometric.

Which a is which. vLLM’s spec_decode metrics report accepted ÷ drafted, not the per-position a above. Tokens per step is then 1 + rate × k (DFlash2 depth 4: 1 + 0.577 × 4 = 3.31, which matches the engine’s own tokens-per-pass). Read 0.45 as a per-position a and the formula undercounts. Under the formula’s assumption, a reported 0.435 at depth 4 corresponds to a per-position a of about 0.69.

With a = 0.45, depth 4 already gets 1.78 of the 1.82 tokens per step that infinite depth would give. With a = 0.80, depth 4 gets 3.36 and depth 8 gets 4.33. Each extra drafted token is worth a^k, so the gain from depth falls off geometrically while the cost of each extra position stays roughly the same.

Why deeper costs

Each step drafts k tokens and verifies k + 1 positions per seat. With one stream, the GPU has spare compute and the extra positions are almost free. With many seats the batch already fills it, and a rejected draft takes throughput from another request. Speculative positions also hold KV.

The reported rate falls as depth rises even when the drafter is no worse per position: depth 4 → 6 took the MTP rate from 0.435 to 0.333, but both work out to a per-position a of about 0.69 (computed under the formula’s assumption). Deeper positions are reached less often, so they pull the average down.

Measured A/B

MTP, depth 4 vs 6. Qwen3.8 Flash-Next MoE (NVFP4), vLLM, TP=2, 8 seats, real prompts, 3 runs each (2026-09-30). Adoption rule fixed before the runs: +10 % on the primary metric or keep the default.

At 16 seats: 183.6 (depth 4, 3 runs) vs 168.3 (depth 6, 1 run). Depth 6 drafts more tokens per pass (3.00 vs 2.74), yet the larger verify batches took more KV, pushed cached prefixes out and cost wall time.

DFlash2, depth 4 vs 7 vs 8. Qwen3.8 27B dense NVFP4, vLLM, TP=2, prefill budget 4,096 (2026-10-03). Depth 4 was run twice; its totals agreed within 1.4 %.

Depth 16 is from a separate run with a different harness. It came in at 116.8 tok/s aggregate at 8 streams, 0.89x of the same harness with speculation off (131.1): slower than no drafter at all.

The best depth depends on concurrency. Depth 8 wins with 1 stream, depth 7 at 4 streams, and depth 4 at 8. The panel’s default is set for the 8+ seats the pair actually serves. A pair serving one user at a time should be set deeper.

Synthetic prompts overstate acceptance

The same MTP profile reported 0.80 on synthetic prompts (lists of NATO-alphabet words) and 0.43–0.47 on prompts from 24 hours of real traffic (2026-09-30). Repetitive synthetic text is easy for a drafter to guess. At depth 4 that is 4.20 vs 2.74 tokens per pass. A depth sweep on synthetic text therefore rewards depth that real traffic does not repay. Every depth above was chosen on real prompts or real Claude Code sessions.

A separate effect of the same kind: DFlash2’s published depth-16 acceptance was 93.1 % on two GB10s, against 14–28 % measured here on a different NVFP4 export of the target (cause not isolated).

Current values

Related settings: concurrency (decides where the depth crossover sits), prefix caching (what deeper drafts pushed out), KV type (FP4 KV and a drafter do not combine on this pair).

vllm settings