Concurrency (max_num_seqs): how many requests decode at once
Result. On 2x DGX Spark (GB10, TP=2), 16 seats decoded +25% more tokens per second in total than 8 seats, each stream ran 34% slower, and on re-sent 15k-token prompts the warm prefix-hit rate fell from 73% to 11%. 8 was kept on 2026-09-30. On an RTX 5090 node, aggregate decode keeps climbing to 9–12 seats.
What the setting is
max_num_seqs = seats: the most requests the scheduler keeps in one running batch. Request 9 on an 8-seat engine waits in the queue, even with KV memory free.
- Continuous batching. Every engine step runs one forward pass over all seated requests. A finished request leaves at the end of a step; a queued one takes its seat on the next. Nobody waits for the batch to drain.
- One weight read per step, shared. A decode step reads every active weight from memory once, whatever the seat count. 8 seats get 8 tokens from that read, 16 seats get 16.
- Why it pays here. Decode on the Spark pair measured 78–85% of the 273 GB/s GB10 bandwidth (peak 230.9 GB/s per GPU), while using 1–7% of the measured 323.2 TFLOP/s NVFP4 ceiling (2026-09-30). The pass is waiting on memory, so extra rows in the batch are nearly free until KV reads or compute catch up.
- Why it does not pay forever. Each seat adds its own KV reads and, on an MoE, wakes more experts. On the 512-expert Flash-Next model, a 40-token batch touches only 55% of the experts; reaching 90% expert saturation needs ~117 tokens per step, about 23 seats (2026-09-28). On SGLang with the dense 27B, a decode step costs ≈ 70 ms + 7.5–8.7 ms per seat: 67 ms at 1 seat, 130 at 8, 205 at 18 (2026-10-03).
Measured: 8 vs 16 seats on real prompts
2x DGX Spark, vLLM, Qwen3.8 Flash-Next (512-expert MoE, NVFP4), prefill budget 8,192, 4 speculative tokens. 100 real user prompts with ~2k tokens of real context each, 512 out, 32 clients. 3 reps per setting, engine argv verified at launch, 0 errors, 0 preemptions.
| 2x DGX Spark, 3-rep mean | 8 seats | 16 seats | Δ |
|---|---|---|---|
| Aggregate decode, tok/s | 147.0 | 183.6 | +25% |
| Per-stream decode, tok/s | 20.3 | 13.4 | −34% |
| Time to first token, s (32 clients queued) | 71.9 | 40.7 | −43% |
| Wall time, 100 requests, s | 348.4 | 278.8 | −20% |
| KV pool used, peak | 16% | 31% | ×2 |
| Warm re-run of 48 × ~15k prompts: prefix hits | 73% | 11% | |
| Warm re-run: wall time, s | 179.6 | 387.6 | ×2.16 |
TTFT falls at 16 because 32 clients drain through twice the seats, not because a step got faster. Synthetic 2k-token prompts the day before gave 16 seats 1.17x (233.5 → 272.5 tok/s); real prompts gave 1.25x.
Why 8 was kept
- Prefix cache. The warm re-run sends the same 48 long prompts again. At 8 seats live KV peaked at 25% of the pool and 73% of prompt tokens came from cache. At 16 seats live KV hit 52%, LRU evicted the cached prefixes, hits fell to 11%, and the run took 2.16x as long. On this pair a 92.6%-hit window decoded ~128 tok/s and a 0%-hit window 6.1 tok/s (2026-09-28): the cache matters more than the seat count. Details: prefix caching.
- Caveat. Re-sending prompts in insertion order is LRU’s worst case. A session-ordered replay was built and never run; an earlier corpus run at 24 seats held 80–83% hits with session locality. Open.
- Decision rule, frozen before the runs: adopt only at ≥ +10% on the primary metric (deep-research mixed wall time) with no secondary metric worse than −10%. Between-generation noise on this pair is 5–10%.
Seats vs the KV pool
Seats are a ceiling, not a reservation. vLLM allocates KV blocks as tokens arrive. If seated requests outgrow the pool, vLLM’s V1 scheduler preempts one and recomputes it later (general vLLM behaviour; 0 preemptions in every run on this page). The panel’s question is whether seats × typical context fits the KV pool:
| Model, engine, 2x DGX Spark | Seats | Pool | Full contexts the pool holds | Result |
|---|---|---|---|---|
| Flash-Next MoE, vLLM, 28 GB pin | 6 | — | 6.52 at 262,144 (vLLM’s own log line) | default since 2026-10-05: memory-safe |
| Flash-Next MoE, vLLM | 24 | — | — | rests at 1.7 GiB free idle, inside the 0.8–4.2 GiB band that hung both boxes; deferred |
| Flash-Next MoE, SGLang | 6 | 1,720,000 | 6.6 at 262,144 | 49.0 tok/s |
| Flash-Next MoE, SGLang | 16 | 1,482,304 | 5.7 | 96.2 tok/s |
| Flash-Next MoE, SGLang | 32 | 1,140,288 | 4.35 | 88.8 tok/s (−7.7%) |
| DeepSeek-V4-Flash NVFP4, vLLM | 8 | 181,777 | 1.39 at 131,072 | serving; 8 seats fit only if requests average ≤ 22,722 tokens |
SGLang rows: 15-min deep-research swarms, 2026-10-05. On that hybrid model every seat reserves ten linear-attention state slots, so the token pool shrinks as seats grow. 32 seats bought queueing and cost cache. Full contexts = pool ÷ window, computed. DeepSeek pool: generation 235, 2026-10-24.
Seats also size vLLM’s CUDA graphs: the largest captured batch is derived as min(seats × decode query length × 2, 512). Raising seats spends graph memory, not KV.
When more seats do not add throughput
Clean-decode windows on the Spark pair do scale with seats: SGLang with the dense 27B and its DFlash2 drafter went 175 tok/s at 8 seats to 290–366 at 25–28 (2026-10-16). The 4-day production log, bucketed by running requests, stays flat at 175–192, and vLLM on the same pair is flat too (127 / 130 / 126 tok/s at 5–8 / 17–24 / 25+). The cause is prefill: its share of wall time grows from 17% to 45% as seats fill, and prefill on GB10 TP=2 costs ~0.49 ms per prompt token. The RTX 5090 node computes prompts at ~5k tok/s, 5–10x cheaper per token, so its seats turn into throughput. See prefill budget.
Defaults in the panel (2026-10-24)
| Model on 2x DGX Spark | Engine | Seats | Basis |
|---|---|---|---|
| Qwen3.8 Flash-Next MoE | vLLM | 6 | what the 28 GB pin holds at 262,144; throughput at 6 not measured |
| Qwen3.8 Flash-Next MoE | SGLang | 16 | 4-point sweep, 2026-10-05 |
| Qwen3.8-27B dense + DFlash2 | SGLang | 32 | 98.4 tok/s over 13,210 requests, 2026-10-03/04 |
| DeepSeek-V4-Flash NVFP4 | vLLM | 8 | recipe value; not swept |
Rule of thumb from these runs: raise seats while decode is bandwidth-bound and the pool still holds seats × typical context plus the prefixes you want cached. On the Spark pair that limit came at 8 for vLLM on long contexts; on the RTX 5090 node it had not come by 12. Related: context length, speculative tokens, memory budget.