KV type: kv_cache_dtype
A smaller KV type multiplies the KV pool’s tokens, if the model’s attention kernel can read it. The flag list is the same for every model. The kernel decides which values actually start. Of the seven launch rows below, six refuse nvfp4 KV, each for its own reason.
The values
kv_cache_dtype | Bytes / element | Tokens per GB vs bf16 | What it is |
|---|---|---|---|
auto | the checkpoint’s | — | whatever the checkpoint and engine resolve; not always bf16 |
bfloat16 / bf16 | 2 | 1.0x | 8 exponent, 7 mantissa bits |
float16 | 2 | 1.0x | 5 exponent, 10 mantissa bits |
fp8_e4m3 | 1 | 2.0x | 4 exponent, 3 mantissa: more precision |
fp8_e5m2 | 1 | 2.0x | 5 exponent, 2 mantissa: more range |
fp8 | 1 | 2.0x | vLLM’s generic name, maps to e4m3; DeepSeek-V4 reads it as fp8_ds_mla |
nvfp4 | 0.5625 | 3.56x | FP4 E2M1 + one FP8 scale per 16 elements |
fp4_mx_block16 (SGLang) | 0.5625 | 3.56x | FP4 + one 1-byte scale per 16 elements |
fp8_ds_mla | 584 B / token / layer | — | DeepSeek-V4’s packed sparse-MLA page |
nvfp4_ds_mla | 352 B / token / layer | — | packed FP4 MLA page, DeepSeek-V3.2 / GLM geometry only |
Bytes per element: vLLM utils/torch_utils.py and SGLang mem_cache/kv_cache_dtype.py, source-read in the images 2026-10-22. 0.5625 = 0.5 + 1/16. Packed MLA pages are stated per token per layer because the layout is not one element type.
auto is not a synonym for bf16. With the same auto, vLLM resolved Qwen3.8 Flash-Next to bf16 on 2x DGX Spark and the Qwen3.8 27B NVFP4 checkpoint to fp8_e4m3 on an RTX 5090. Read the resolved type in the startup log before you compare pools.
fp8_e5m2 has not been measured on any machine here.
FP8 vs NVFP4, measured
Measured 2026-10-21/22, written up in KV cache: FP8 vs NVFP4: 1.64x the tokens on an RTX 5090 (vLLM), 1.60x on 2x DGX Spark (SGLang), against 1.78x in theory. On the Sparks FP4 KV ruled out the speculative drafter, which was worth more.
Who refuses what
| Model · engine · GPU | Launches with | Refused | Why |
|---|---|---|---|
| Qwen3.8 27B dense · vLLM · RTX 5090 (sm_120) | nvfp4 since 2026-10-22 | — | FlashInfer FA2 reads NVFP4 pages on sm_80/90/120 |
| Qwen3.8 Flash-Next · vLLM · GB10 (sm_121) | bf16 | nvfp4 | only the SM100 trtllm-gen FlashInfer path reads it: “requires the SM100 trtllm-gen FlashInfer path” |
| Qwen3.8 Flash-Next · SGLang · GB10 | bf16 | fp8_e4m3, fp8_e5m2, nvfp4, fp4_mx_block16 | its sparse-attention kernel on SM121 takes BF16 KV only |
| Qwen3.8 27B dense · SGLang + drafter · GB10 | fp8_e4m3 | nvfp4, fp4_mx_block16 | dies with the drafter (gen 214–217) or serves wrong output (gen 219) |
| GLM-5.3-Flash · vLLM · GB10 | fp8_e4m3 | nvfp4 | MLA: this vLLM refuses an nvfp4 cache with MLA |
| DeepSeek-V4-Flash · vLLM · GB10 | fp8 (= fp8_ds_mla) | fp8_e4m3, nvfp4 | its sparse MLA takes auto, fp8_ds_mla, fp8 only |
| any · vLLM 0.19.1 ROCm · AMD iGPU | bf16 | nvfp4 | no FP4 KV type in this build at all |
vLLM rows: source-read in each image, 2026-10-22/23. SGLang rows and the RTX 5090 row: run.
fp8_e4m3 and fp8 are both 1 byte of e4m3, but DeepSeek-V4 accepts only the second: the name selects the packed page, not just the element type.
DeepSeek-V4: nvfp4 KV is not a flag
DeepSeek-V4-Flash’s attention has its own cache writer and reader, and both work on the 584 B fp8_ds_mla page. Neither has an nvfp4 path. nvfp4_ds_mla exists in vLLM, but for the DeepSeek-V3.2 / GLM attention geometry, not V4’s. Getting FP4 KV on this model means new attention kernels.
Even then it is the smaller lever. 352 B against 584 B is 40 % less per entry. The cache layout is worth about 9x (see KV pool).
Choosing
- Check the refusal first: a refused type fails at start, or worse, starts and answers wrong (SGLang gen 219: 17×23 answered as “22 by 5”).
- nvfp4 pays where nothing else depends on the pool’s format: the RTX 5090 node (no drafter, 1.64x).
- Where decode is bound by reading weights, a smaller KV type buys capacity, not speed. On the Spark pair weights were 81 % of the bytes per decode pass.
More settings: vLLM settings, explained. Background on the FP4 formats: MXFP4 vs NVFP4.