KV pool: kv_cache_memory_bytes vs gpu_memory_utilization
The KV pool is a number of tokens, not a number of gigabytes. vLLM divides the bytes you give it by what one token costs in every attention layer. Bytes per token range from 5,987 (modeled) to 55,011 (measured) across the models on this panel, so the same 10 GB can hold 181,777 tokens or 1.67 million.
Two ways to size it
gpu_memory_utilization | kv_cache_memory_bytes (a pin) | |
|---|---|---|
| What you state | fraction of the card vLLM may use in total | the KV pool itself, in bytes, per GPU |
| KV pool | what is left of the fraction after weights, activations, CUDA graphs | exactly the pin |
| Changes when | weights, the prefill budget, graph sizes or seat count change | never, unless you change it |
| Start check | vLLM refuses a start if fraction × card > free memory | the same check still runs on the fraction |
The pin wins on a shared or unified-memory box: the pool stays the same size from one start to the next, and the rest of the budget is the operator’s to read.
The fraction still matters with a pin. On the RTX 5090 node vLLM refuses any start where fraction × 31.36 GiB exceeds the 30.85 GiB free at start, so 0.99 and 1.0 can never start; 0.9837 is the ceiling (measured 2026-10-11/12). That node runs a 7.92 GiB pin (8,501,810,176 B) beside a fraction of 0.98.
Reading the startup line
vLLM prints two figures:
GPU KV cache size: 181,777 tokens
Maximum concurrency for 131,072 tokens per request: 1.39x- Tokens = pool bytes ÷ bytes per token. 10e9 B ÷ 55,011 B = 181,777.
- Max concurrency = tokens ÷
max_model_len. 181,777 ÷ 131,072 = 1.39. It counts full-length requests only: 1.39 requests at 131k, or many more short ones. It is not the seat count (concurrency is a separate setting). - On a TP=2 pair both figures are per rank.
A second case, 2026-09-30: Qwen3.8 Flash-Next on the same pair, 28e9 B pin, 262,144 context, 4 speculative tokens. vLLM reported 1,708,119 tokens and 6.52x. At 3 and 2 speculative tokens the same pin gave 1,752,822 and 1,791,558: the drafter’s KV comes out of the same pool.
Bytes per token, by model
| Model, engine, KV type | B / token / rank | Tokens in 10 GB | Source |
|---|---|---|---|
| DeepSeek-V4-Flash NVFP4, vLLM, fp8_ds_mla | 55,011 | 181,777 | measured, gen 234/235, 2026-10-24 |
| Qwen3.8 27B dense NVFP4, vLLM, bf16 | 36,860 | 271,297 | measured, gen 103/104, 2026-10-02 |
| Qwen3.8 27B dense NVFP4, SGLang, fp8 (target + drafter pools) | 21,502 | 465,073 | measured, gen 123, 2026-10-03 |
| Qwen3.8 Flash-Next MoE NVFP4, vLLM, bf16 | 16,392 | 610,054 | measured, 2026-09-15 / 09-30 |
| GLM-5.3-Flash NVFP4, vLLM, fp8 | 6,973 | 1,434,103 | a published 2x DGX Spark run, not measured here |
| DeepSeek-V4-Flash, compressed layout | ~5,987 | ~1,670,000 | modeled, not run |
"Tokens in 10 GB" = 10e9 ÷ bytes per token, arithmetic. Only the DeepSeek row was run at 10e9.
DeepSeek-V4-Flash on 2x DGX Spark
nvidia’s NVFP4 checkpoint, vLLM nightly 2026-10-15, TP=2, 8 seats, 131,072 context, prefill budget 8,192, drafter off.
| Gen | Pin | Pool, tokens | Full 131,072 contexts | Outcome |
|---|---|---|---|---|
| 234 | 16e9 B (14.90 GiB) | 290,858 | 2.22 | killed in warmup: 7.9 GiB available, under the 8 GiB floor |
| 235 | 10e9 B (9.31 GiB) | 181,777 | 1.39 | serving; init 357 s; 28.8 tok/s at 1 stream |
| 237 | 10e9 B | 181,777 | 1.39 | serving |
The plan, written before the first start, expected 13,952 B/token, read off two published 2x DGX Spark pools of the same model. At that figure 16e9 B would have held ~1.15 M tokens, enough for all 8 seats at full length. The measured figure is 3.9x larger.
Where the memory went, per rank
On a DGX Spark the GPU and the OS share one 128 GB LPDDR5X pool. A start that allocates a bigger KV pool takes it from the same memory the OS, the page cache and the warmup kernels need. The start’s memory guard kills a launch if available memory stays under 8 GiB for 3 polls 2 s apart, before the engine serves; a healthy serving pair settles near 4.8 GiB, so the floor applies only during the start. Gen 234 hit 7.9 GiB in the b12x warmup and was killed before it could serve.
With the drafter on, the start estimate needs 115.1 GiB at a 10e9 pin against ~114.5 GiB free: the pin has to drop to 9e9 or less (8e9 leaves a margin). The fuller budget is on memory budget.
Why 55,011 bytes
Analysis, not a log line. DeepSeek-V4’s attention is designed to keep most of its KV compressed along the sequence. The measured 55,011 B/token matches caches allocated at full length and uncompressed for every layer. With the design’s hybrid, compressed layout the same model would need about 5,987 B/token, which at the same 10e9 B pin is about 1.67 M tokens, 12.7 full 131,072 contexts. That figure is modeled from the architecture; no build here has run it. Gen 237 logs no hybrid-allocator or padding line at INFO, so the log cannot confirm the cache groups either way.
The pool is the lever with the largest headroom on this model, and it is a layout change in the engine, not a flag. A smaller KV type saves at most 40 % per token here (KV type).
Sizing it
- Find bytes per token: the startup line’s tokens ÷ the pool bytes. Expect the published figure to be wrong for your build (3.9x off here).
- Tokens needed = seats × typical context, not seats ×
max_model_len, unless every seat really runs full length. Prefix-cache hits also live in this pool (prefix caching). - On unified memory, size from available memory during warmup, not at idle. On this pair idle and loaded differ: ~11 GiB free just after a start vs ~4.8 GiB under a saturating load (Qwen3.8 Flash-Next, 2026-09).
- A pool larger than one full context is the floor: below it a single
max_model_lenrequest cannot be admitted (context length).
More settings: vLLM settings, explained. Overflow beyond the pool: KV offload.