vLLM settings, explained
Ten settings, one page each. Every value in the last column is from the running launch: DeepSeek-V4-Flash-0731 NVFP4, TP=2 on 2x DGX Spark (GB10), vLLM nightly (2026-10-15 build), generation 237, 2026-10-24.
| Setting | vLLM flag | Trades | DeepSeek-V4-Flash, gen 237 |
|---|---|---|---|
| KV pool | --kv-cache-memory | cached tokens vs memory for everything else | 10e9 B (9.31 GiB) a rank = 181,777 tokens |
| KV type | --kv-cache-dtype | bytes per token vs precision and kernel support | fp8 (this model’s attention takes no other) |
| KV offload | --kv-transfer-config (LMCache) | evicted prefixes kept on disk vs reload latency | off |
| Prefix caching | --enable-prefix-caching | nothing extra; reuse lasts until the pool needs the blocks | on, 256-token blocks; 88.4 % hits |
| Speculative tokens | --speculative-config num_speculative_tokens | tokens per target pass vs wasted verify work | 0 (recipe: 5) |
| Drafters | --speculative-config method | acceptance vs drafter memory | DSpark in the checkpoint, not loading yet; 5.06 GiB a rank by its shard headers |
| Concurrency | --max-num-seqs | aggregate tok/s vs per-stream speed and pool share | 8 |
| Prefill budget | --max-num-batched-tokens | prefill speed vs decode stalls and activation memory | 8,192 |
| Context length | --max-model-len | longest request vs how many full ones the pool holds | 131,072 (model: 262,144) |
| Memory budget | none: the sum of the above | whether a start fits at all | 102.0 GiB held a rank; start needs estimate + 8 GiB free |
Flags and values: the head rank's launch arguments, 2026-10-24. Hit rate: the engine's stats line at 14:50 UTC. 102.0 GiB: available memory before vs after generation 235, same arguments.
How they connect
- Memory first. Weights (79.29 GiB a rank), the drafter, the KV pool and 13.4 GiB of overhead must fit one GB10’s shared memory with 8 GiB to spare. The KV pool is the term that moves.
- The pool is shared by three settings. Concurrency × context length is what the pool is promised (8 × 131,072 = 1,048,576 tokens, 5.8x the 181,777 it has); prefix caching lives in whatever is left; KV type sets how many tokens a byte holds.
- Two settings decide decode speed. Speculative tokens and the drafter set tokens per weight read; concurrency spreads one weight read over more streams. Decode on this pair is memory-bandwidth bound (78–85 % of 273 GB/s, measured 2026-09-30 on an earlier model).
- One setting decides prefill vs decode. The prefill budget sets how many prompt tokens one step takes, and so how long decode streams wait behind a new prompt.
Measured single-stream speed of this launch: 28.8 generated tok/s with thinking off (generation 235, 2026-10-24). Same-load sweeps of these settings on DeepSeek-V4-Flash: not run yet; 8 seats and 8,192 are the published recipe’s values.