KV offload: a RAM or disk tier behind the KV pool
An offload tier is a second, slower cache for prompt KV. When the GPU’s KV pool is full, vLLM evicts blocks. With a tier the evicted blocks are already in host RAM or on disk, and a request that comes back with the same prefix copies them in instead of running prefill again. It pays when prefixes return: agent sessions resending a long history, preempted requests resuming.
Two tiers in the panel
| RAM tier | Disk tier | |
|---|---|---|
| Panel setting | kv_offload_ram_gib | kv_offload_gib |
| vLLM flags | --kv-offloading-size N --kv-offloading-backend native | LMCache connector, disk-only config |
| Where | discrete-GPU nodes (RTX 5090) | the 2x DGX Spark pair |
| Bound | the box’s /dev/shm (47.1 GiB on the RTX 5090 node; set to 32) | NVMe space |
| Status | in service from 2026-10-12 | never started with it |
| SGLang | — | not offered: SGLang is launched without the connector |
RAM tier: how vLLM uses it
- Stores are write-through: a block is copied to RAM as it is written, not at eviction.
- A preempted request’s prompt KV is reloaded into its own GPU blocks before it runs again.
- Attention never reads from RAM. Every token decoded still reads KV from the GPU pool.
- The tier is LRU: it frees a chunk only to make room for another, so after enough traffic it is full. On the RTX 5090 node 98.4 GB had been stored into the 32 GiB tier by 2026-10-12.
vLLM’s own usage gauge for the tier counts only chunks pinned by copies in flight, so it reads 0.0 % on a full tier. The panel derives fill as min(tier size, bytes stored since start) and labels it an upper bound.
Measured: RTX 5090 node, 32 GiB RAM tier
Qwen3.8 27B NVFP4, vLLM 0.31.1rc1, 16 seats, FP8 KV at the time, real Claude Code traffic, 2026-10-11/12. Before vs after the tier, not a controlled same-prompt A/B.
| Value | |
|---|---|
| Prompt tokens computed by prefill | 60 % → 12 % |
| Decode per stream, p50 | 15 → 49 tok/s |
| Reloaded RAM → GPU | 467 GiB |
| Copy speed, reload · store | 13.4 · 13.1 GB/s |
| GPU memory free under load | ~674 MiB |
| Without the tier (fraction 0.95, no pin) | pool full at ~8 parallel 10–20k-token requests, 12 preemptions |
Copy speed = bytes ÷ CUDA-event seconds since engine start, read 2026-10-12. Decode rose because less GPU time went to recomputing prefill, not because a pass got cheaper.
For scale, arithmetic: at the 36,166 B/token measured for FP8 KV on this card, 32 GiB holds ~950,000 tokens, about 4x the 235,074-token GPU pool a 7.92 GiB pin gave on FP8.
Cost. The tier takes host RAM (32 GiB of shared memory) and copy bandwidth over PCIe, and GPU free memory under load fell to ~674 MiB. The operator kept it.
Why unified memory refuses a RAM tier
On a DGX Spark the host RAM and the GPU’s memory are one LPDDR5X pool. A RAM tier there would take its bytes from the same memory the KV pool lives in, to cache that pool. The panel refuses it on any unified-memory box.
The disk tier is the option for those boxes: LMCache with local_cpu: false, max_local_cpu_size: 0, a local NVMe directory, 256-token chunks. vLLM’s built-in --kv-offloading-backend lmcache was not used: it wires to a standalone LMCache server whose storage accepts only cpu (LMCache 0.5.4 has the disk branch left as a TODO).
Not proven. On 2026-10-01 the config parsed and the NVMe was writable from inside the container. The engine has never been started with the disk tier, its cost with a zero-size CPU backend is not measured, and the setting defaults to off. Disk speed, hit rate and effect on prefill: not measured.
When it pays
- Prefix reuse is the whole benefit. On the Spark pair the prefix cache alone swung decode from 6.1 tok/s at 0 % hits to ~128 tok/s at 92.6 % hits (2026-09-28). A tier extends the same effect past the GPU pool’s size (prefix caching).
- It helps most when the pool is small against the working set: ~235k tokens of GPU pool under 16 seats of 10–20k-token agent sessions.
- It does nothing for a single long request that fits, and nothing for decode speed per pass.
More settings: vLLM settings, explained. Pool size first: KV pool; bytes per token: KV type.