Context length (max_model_len): the per-request token cap
Result. max_model_len is a cap, not a reservation. Full-length requests that fit = KV pool ÷ max_model_len: DeepSeek-V4-Flash on 2x DGX Spark has a 181,777-token pool, so 1.39 requests at 131,072, the operator’s default cap since 2026-10-13. Shorter requests share the same pool, so 8 seats still work while requests average ≤ 22,722 tokens.
What it caps
- Per request: prompt + generated tokens. A longer prompt is rejected; generation stops at the cap (general vLLM behaviour).
- At start: vLLM refuses to start if the KV pool cannot hold one request of
max_model_len(general vLLM behaviour; not tried on this pair). - What it does not do: reserve memory per seat. vLLM allocates KV blocks as tokens arrive, so a 4k request uses 4k tokens of pool whatever the cap.
- The number vLLM logs: “Maximum concurrency for N tokens per request: X x” is pool ÷ N, a worst case that assumes every request runs to the cap.
Full contexts vs the limit, one fixed pool
The pool here is small on purpose: a 16e9-byte pin (290,858 tokens) was killed by the memory guard at warm-up with 7.9 GiB available, so the pin dropped to 10e9. To raise the cap past 181,777 on this model you need a bigger KV pool, not a bigger number in this field.
Fixed per-request state
Hybrid models (linear-attention layers next to full attention) also keep fixed state per request, so the worst case is not a clean division:
| Node | Pool | Per request at 131,072 | Logged max concurrency |
|---|---|---|---|
| RTX 5090 node, vLLM, 27B NVFP4, FP8 KV | 116 blocks × 1,568 tokens | 84 blocks (⌈131,072 ÷ 1,568⌉) + 6 fixed = 90 | 1.29 = 116 ÷ 90 |
From the node's own cache_config_info, 2026-10-11. A plain division would say 1.39. On SGLang the same effect hits seats: each seat of the Flash-Next MoE reserves ten linear-attention slots, so the token pool fell from 1,720,000 at 6 seats to 1,140,288 at 32 (2026-10-05).
What a longer limit costs
| Cost | Measured |
|---|---|
| One request can hold more of the pool | a 131,072-token request = 72% of the 181,777-token pool (computed) |
| Prefill time of a full-length prompt | 2x DGX Spark, SGLang: 131k cold = 16 chunks of 8,192 ≈ 55 s, all decode frozen (2026-10-03). RTX 5090 node: 87,119 tokens in 19.2 s, queue included (2026-10-22) |
| The pool must hold one full request | 262,144 does not fit the DeepSeek pool above |
| Seat estimates turn conservative | the panel’s recommender used ⌊pool ÷ max_model_len⌋ at 262,144; the real mean context was ~72k, so it undercounted ~3.6x (2026-09-28) |
| Decode speed | none found: Flash-Next on SGLang, 16 seats at 262,144 = 96.2 tok/s vs 6 seats at 131,072 = 67.4 (2026-10-05). Its sparse attention (indexer budget 2,048) made decode nearly independent of context in a 2026-09-28 audit |
The Flash-Next sweep is the one direct test: 6 seats at 131,072 beat 6 at 262,144 (67.4 vs 49.0 tok/s), but 16 seats at 262,144 beat both. The gap at 6 seats came from too few seats, not from the window. A model with full attention in every layer reads the whole KV each step and would pay per token of context; none was measured here.
The 131,072 cap
On 2026-10-13 the operator capped every model’s default at 131,072 for future deployments, from 262,144. The seats, budget and drafter depth tuned at 262,144 were left unchanged. No measurement is attached to that choice. Defaults on 2026-10-24: 131,072 for Flash-Next (vLLM and SGLang), the dense 27B with DFlash2 and DeepSeek-V4-Flash; 262,144 for GLM-5.3 Flash.
Set it to the longest request you actually serve, then check that pool ÷ limit ≥ 1 and that seats × your average context fits. Related: concurrency, prefill budget, KV type, KV offload.