Prefill budget: tokens per engine step

Result. A bigger budget is not a faster engine. On 2x DGX Spark (GB10, TP=2), doubling it from 4,096 to 8,192 made prefill 64% faster and decode 8.2% slower on a dense model; doubling 8,192 to 16,384 on an MoE made prefill itself 24% slower. Panel defaults on the pair are 2,048–8,192.

What one step holds

max_num_batched_tokens = the token budget of one engine step (one forward pass). How vLLM V1 fills it (general vLLM behaviour, source-read):

1. Decode first. Each seated request in decode takes 1 token, or 1 + K with K speculative tokens. 8 seats at K=4 = 40 tokens.
2. Prefill fills the rest. Waiting prompts are cut into chunks that fit the remaining budget (chunked prefill). A 32k prompt at a 8,192 budget takes 4+ steps.
3. One pass, one clock. Decode and the prefill chunk run in the same pass, so decode never stops. But every stream’s next token waits for the whole step, and a step lasts as long as its chunk takes.

So the budget trades TTFT (fewer, bigger chunks per prompt) against inter-token latency for everyone already decoding (longer steps).

long_prefill_token_threshold

Caps the tokens a single prompt may take in one step, so a giant prompt cannot claim the whole budget and several prompts prefill side by side (vLLM; 0 = off). The panel refuses a threshold above the context length. SGLang has no per-request cap; its budget bounds every chunk. Not measured: every perf-log minute recorded so far ran with it off.

Measured A/Bs

Both on 2x DGX Spark, vLLM, everything else held, engine argv verified at launch.

• The dense model gets the prefill speed-up the arithmetic predicts, and decode pays: longer steps, and the drafter accepts less. vLLM warns on every start to raise the budget for the draft slots. Measured, that advice is wrong on this pair; cutting K is the right half of the warning. See speculative tokens.
• The MoE got slower at prefill too: 52.8 → 44.8 TFLOP/s per GPU. An MoE-only roofline predicted the opposite; it is wrong for this hybrid model. Cause not isolated.
• 4,096 was not tried in the controlled series. Production traffic put it first for Flash-Next, and it is that model’s default now.

SGLang: the budget is also a freeze

SGLang on the pair runs prefill first: when a prefill batch is ready it runs alone (prefill_decode_interval 0, mixed chunks off), and every seated stream emits nothing until it ends. With first-come-first-served admission, an in-flight chunk uses the whole budget, so nothing else is admitted until the prompt’s last chunk.

• Small budget won on SGLang. At 2,048 the pair ran a full day: 98.4 gen tok/s over 13,210 requests (2026-10-03/04). 8,192 measured 119.8, but over only 301 requests in 35 spans, a window the freeze audit was written from. 2,048 is the default.
• The real limit is GB10 prefill. The operator’s correction, confirmed in the perf logs: vLLM on the same pair is just as flat under load (127 / 130 / 126 tok/s at 5–8 / 17–24 / 25+ running). ~0.49 ms per prompt token at TP=2 is the cost, whichever scheduler spends it. An RTX 5090 node on vLLM computes prompts 5–10x cheaper per token.
• Proposed, not applied: --prefill-decode-interval 1 with an 8,192 chunk (estimated +10–20% at 17+ seats) and --enable-mixed-chunk (estimated +3–6%).

What a prompt costs on the pair

Time to first token, SGLang, uncached prompts, 14–15 other requests running, 2026-10-03:

FlashInfer prefill for the linear-attention layers; Triton was 5–8% slower. Single run each.

Defaults in the panel (2026-10-24)

On this hardware the budget that wins is the smallest one that keeps prefill from starving: decode is what most of the traffic waits on. Related: concurrency, context length, prefix caching.

vllm settings