KV cache: FP8 vs NVFP4
Result as of 2026-10-22. NVFP4 KV gives 1.6x the tokens of FP8 in the same memory on both machines measured. On the RTX 5090 node (no speculative decoding) that is the whole trade. On the two DGX Spark boxes it costs the speculative drafter, and the drafter is worth more. Same-load speed benchmarks have not been run yet (see Pending).
The two formats
| FP8 (e4m3) | NVFP4 | |
|---|---|---|
| Value | 8-bit float, 4 exponent / 3 mantissa bits | 4-bit float E2M1 |
| Scale | per tensor / layer | one FP8 scale per 16 elements |
| Bytes per element | 1 | 0.5625 (0.5 + 1/16) |
| Bytes per head, per token | head_size | head_size//2 + head_size//16 |
| Tokens per GB, vs FP8 | 1.00x | 1.78x (theory) |
Page layout: vLLM torch_utils, source-read. 1.78x = 1 ÷ 0.5625.
Weights vs KV
- Weights are quantized once, offline. K and V are produced per request and quantized on the fly as they are written.
- On both stacks here (GB10 sm_121, RTX 5090 sm_120) attention runs in BF16 after dequantizing the cache. FP4 KV buys capacity and decode bytes read, not attention FLOPs.
- General background, not measured here: softmax exponentiates QKᵀ, so an error in K grows in the attention weights. That is why KV formats are checked for accuracy at depth, not just at short prompts.
Measured pool sizes
| Machine | KV | Pool, tokens | Bytes / token | Tokens / GB | vs FP8 |
|---|---|---|---|---|---|
| RTX 5090, 8.5 GB pin | FP8 (to 2026-10-21) | 235,074 | 36,166 | ~27,650 | 1.00x |
| RTX 5090, 8.5 GB pin | NVFP4 (from 19:30Z) | 386,194 | 22,014 | ~45,400 | 1.64x |
| 2x DGX Spark, 47.30 GB | FP8 + drafter (gen 213/220) | 2,887,228 | 16,384 | ~61,000 | 1.00x |
| 2x DGX Spark, 20.27 GB | NVFP4, no drafter (gen 218) | 1,979,584 | 10,240 | ~97,700 | 1.60x |
Both below the 1.78x theory. RTX 5090: likely paging overhead, not checked. Sparks: the NVFP4 pool also holds a one-layer FP8 dequant workspace. Compare within one machine; engines, per-rank layouts and pins differ across them.
On the RTX 5090 the same pin holds 2.39 full-length (98,304-token) requests on FP8 and 3.93 on NVFP4.
RTX 5090 node: NVFP4 in service
Qwen3.8-27B NVFP4 weights, vLLM 0.31.1rc1, no speculative decoding. vLLM 0.31.1 on sm_120 reads NVFP4 KV through FlashInfer FA2.
| Check on NVFP4 KV (2026-10-22) | Result |
|---|---|
| Arithmetic | correct |
| Tool call | correct |
| Planted code recalled at 8k / 60k / 75k depth | 3 of 3 |
| Prefill, 87,119-token prompt | 19.2 s ≈ 4,526 tok/s (incl. queue) |
| Single-stream decode (one other stream running) | 60.3 tok/s |
| FP8 decode on this card, same load | not measured |
2x DGX Spark: FP4 KV vs the drafter
GB10 x2, TP=2, SGLang 708f51e, Qwen3.8-27B NVFP4 + DFlash2 speculative drafter, 32 seats, 131,072 context.
| Gen | KV | Drafter | Outcome |
|---|---|---|---|
| 213 | FP8 | on, FlashInfer | serves; all checks pass |
| 214 | NVFP4 | on, FlashInfer | dies at init: FlashInfer cannot decode NVFP4 |
| 215 | NVFP4 | on, trtllm_mha | dies at CUDA graph capture: “supports decode only”, the drafter’s block pass is not |
| 216 | MX-FP4 | on, Triton | killed by the memory guard: pool at the fraction’s max, large allocation at drafter init |
| 217 | MX-FP4, 28 GB pin | on, Triton | dies at first batch: the DFLASH writer copies BF16 into the FP4 pool |
| 218 | NVFP4, 28 GB pin | off | serves; all checks pass; 1,979,584-token pool |
| 219 | NVFP4 | on, own FP8 pool | starts (4,074,176 tokens, 1.41x the FP8 pool) but serves wrong output |
| 220 | FP8 | on | rollback; all checks pass |
Gen 219 used SGLang’s --speculative-draft-kv-cache-dtype to give the drafter an FP8 pool of its own. It answered 17×23 as “22 by 5”, made no tool call and missed the planted code at 8k, 60k and 110k. Drafter acceptance looked healthy (accept length 1.6–3.9). Suspect, not proven: the target’s verify pass runs on the prefill backend and misreads the NVFP4 pool. The launcher now refuses FP4 KV together with a drafter.
Why the drafter wins on this pair
Decode on the pair is weight-bandwidth bound: each pass reads the weights once. With the drafter a pass yields several accepted tokens per stream; without it, one. At 2–6 streams that is ~2.3x per stream in favour of FP8 + drafter, while NVFP4 only shrinks the 35 GB/s KV share.
Who serves NVFP4 KV
| Engine | GPU | NVFP4 KV | Note |
|---|---|---|---|
| vLLM 0.31.1rc1 | RTX 5090 (sm_120) | serves | FlashInfer FA2; in service |
| vLLM | GB10 (sm_121) | refused | parser accepts; served only on CC 10.x (FlashInfer trtllm-gen) |
| vLLM 0.19.1, ROCm | AMD | no | no FP4 KV at all |
| SGLang 708f51e | GB10, no drafter | serves | gen 218 |
| SGLang 708f51e | GB10, DFlash2 drafter | no | dies (214–217) or wrong output (219) |
vLLM rows: source-read from the images, except the RTX 5090 row (run). SGLang rows: run on the pair.
Pending: benchmarks
Not run yet. Each cell will be the same prompt set at the same concurrency on the same machine, FP8 vs NVFP4.
| Metric | RTX 5090 FP8 | RTX 5090 NVFP4 | Sparks FP8 + drafter | Sparks NVFP4, no drafter |
|---|---|---|---|---|
| Decode tok/s per stream, 1 stream | — | — | — | — |
| Decode tok/s per stream, 4 streams | — | — | — | — |
| Decode tok/s per stream, 8 streams | — | — | — | — |
| Prefill tok/s, ~9k prompt | — | — | — | — |
| Prefill tok/s, ~70k prompt | — | — | — | — |
| Recall of a planted code at 8k / 60k / 110k | — | — | — | — |
| Answers matching FP8 on a fixed eval set | — | — | — | — |
Spot figures above (60.3 tok/s, 4,526 tok/s, ~32 / ~14 tok/s) were taken under whatever traffic was running and are not entered here.
FP8 vs NVFP4 KV, as of 2026-10-22: NVFP4 on a node without speculative decoding (1.64x pool, checks pass); FP8 on a decode-bound pair that runs a drafter, until an SGLang build serves FP4 KV with one. Background on the FP4 formats: MXFP4 vs NVFP4; on the attention backends: FlashInfer or Triton for local inference.