Prefix caching (--enable-prefix-caching)
Result, 2026-10-24. DeepSeek-V4-Flash NVFP4 on 2x DGX Spark (GB10), generation 237: prefix cache hit rate 0 % → 75.2 % in the first 110 s of agent traffic, 88.4 % at the end of the third hour. At 88.4 %, 11.6 % of prompt tokens go through prefill: 8.6x less prefill work than the same traffic with the cache off (computed: 1 ÷ 0.116).
What it does
| Flag | --enable-prefix-caching (on by default in this vLLM build; we pass it anyway) |
| Unit | one KV block = 256 tokens here (--block-size 256) |
| Key | hash of the block’s tokens plus the hash of the block before it (sha256 in this build) |
| Hit | a new request’s leading blocks hash the same: their KV is reused, prefill starts after them |
| Miss | anything after the first differing token, and the last, partly filled block |
| Memory | none extra: cached blocks are pool blocks no request is using |
| Eviction | least-recently-used free block, when a running request needs a new one |
Defaults and hash read from the serving image's config/cache.py, 2026-10-24. Block chaining and LRU eviction: vLLM's automatic-prefix-caching design.
Because each block’s hash includes its parent’s, one changed token early in the prompt makes every later block a miss. A cache hit needs the same tokens in the same order from the first token on.
Why agents hit it
A coding agent re-sends its whole conversation every turn: system prompt, tool definitions, the history, then the new message. Only the tail is new. On this pair’s earlier SGLang serving, the first ~39k tokens were identical across agent sessions: 26 of 244 single-sequence prefills matched exactly that shared prefix and nothing more (audit, 2026-10-03). A new session pays for its system prompt once per pool, not once per turn.
Generation 237, hit rate over time
| UTC | hit rate | prompt tokens computed per 100 sent |
|---|---|---|
| 11:24:40 | 0.0 % | 100 |
| 11:26:30 | 75.2 % | 24.8 |
| 11:54:40 | 84.3 % | 15.7 |
| 14:50:21 | 88.4 % | 11.6 |
Hit rate: hit tokens ÷ queried tokens over the most recent 1,000 requests (CachingMetrics(max_recent_requests=1000) in the image). Third column computed as 100 × (1 − hit rate).
What a hit is worth in throughput
| Measurement on this pair | hit rate | generated tok/s |
|---|---|---|
| Qwen3.8 MoE on vLLM, production traffic (2026-09-28) | 92.6 % | ~128 |
| Same engine, 72k-token synthetic corpus, unordered (2026-09-28) | 0 % | 6.1 (3,334 ms per step) |
| Qwen3.8 MoE, working set < 0.8 of the pool (2026-10-02) | 89 % | 160 |
| Same, working set > 1.2 of the pool | 52 % | 49 |
| DeepSeek-V4-Flash, cache off vs on | — | not measured |
Rows 1–2: 21x apart, but not an A/B; the 0 % run used a synthetic corpus with a 357:1 prompt-to-output ratio. Rows 3–4: binned production minutes, from why a sparse MoE leaves GPU compute idle.
Prefill on this pair is compute-heavy and decode is bandwidth-bound; every prompt token served from the cache is a prefill step the decode streams do not wait behind.
Eviction: seats × context vs the pool
Cached blocks live in the same pool as running requests. When the requests need the room, the least recently used cached blocks go first, and the next turn of that session prefills from scratch.
| DeepSeek-V4-Flash, generation 237 | value |
|---|---|
| Pool, a rank | 181,777 tokens = 710 blocks of 256 |
| Worst case: 8 seats × 131,072 tokens | 1,048,576 tokens = 5.8x the pool |
| Highest KV usage logged, 11:24–14:50 | 74.3 % (14:40:01, 4 running) |
| Preemptions in that window | not counted |
Pool: engine startup log. Blocks and 5.8x computed. KV usage: the stats lines charted above.
The pool is fixed at start, so what keeps the hit rate up is fewer long sessions sharing it, or a bigger pool. On the Qwen3.8 pair the hit rate held at 86–89 % until demand passed the pool, then fell to 64 % and 52 % (table above). The pool’s size and what it costs: KV pool; seat count: concurrency; per-request length: context length. A disk tier that keeps evicted blocks: KV offload.
Part of vLLM settings, explained.