FP4 KV cache on the local fleet: 1.6x the pool, benchmarks pending
Status 2026-10-22: KV cache switched from FP8 to NVFP4 where it works. Throughput benchmarks are still pending: no same-load FP8 vs NVFP4 speed comparison has been run yet.
| RTX 5090 node | 2x DGX Spark | |
|---|---|---|
| Engine | vLLM 0.31.1rc1 | SGLang 708f51e, TP=2 |
| Speculative decoding | none | DFlash2 drafter |
| NVFP4 pool vs FP8 | 1.64x (386,194 vs 235,074 tokens) | 1.60x per GB (drafter off) |
| Checks on NVFP4 (math, tool call, recall at depth) | pass | pass without drafter; wrong output with it |
| KV in service tonight | NVFP4 | FP8 + drafter |
| Same-load FP8 vs NVFP4 speed | pending | pending |
- The 5090 node has no drafter to lose, so NVFP4 is a larger pool with no cost measured so far: 3.93 full-length requests fit instead of 2.39.
- On the Sparks, eight starts in one day: FP4 KV with the drafter died four ways and, with the drafter on its own FP8 pool, served wrong answers. NVFP4 without the drafter decoded ~14 tok/s per stream vs ~32 for FP8 + drafter (different loads), so the pair rolled back to FP8.
- The launcher now refuses FP4 KV together with a drafter.
Pool sizes, the eight starts, charts and the list of pending measurements: KV cache: FP8 vs NVFP4. The attention-backend side: FlashInfer or Triton for local inference.