vLLM cluster throughput calculator — any box, any link, any model
A single-file calculator for vLLM serving speed on hardware you compose: N boxes × (GPU × count), the GPU-to-GPU link inside a box, the NIC and fabric between boxes. Load any of 30 catalogued models (or type your own), pick the weight and KV format, TP / PP / DP / EP, max_model_len, gpu_memory_utilization and MTP depth, and get the memory map per GPU, decode tok/s per request and aggregate against context for concurrency 1 … 256, prefill tok/s and TTFT against prompt length, the step-time breakdown (memory · compute · network · overhead) and a ranked comparison of every preset configuration.
Calibration
Thirteen measured points, three machines. Residual = modelled ÷ measured − 1 with the shipped constants.
| setup | measured | modelled | err |
|---|---|---|---|
| 1× DGX Spark · Qwen3.8-27B NVFP4 · no spec · depth 0 | 11.5 tok/s | 11.5 | 0 % |
| 1× Spark · 27B · MTP 3 · depth 0 / 32K / 100K | 18.8 / 20.4 / 18.9 | 21.7 / 20.0 / 17.1 | +16 / −2 / −9 % |
| 1× Spark · 27B · prefill pp2048 | 2,083 tok/s | 2,110 | +1 % |
| 1× Spark · Qwen3.8-Flash-Next NVFP4 · MTP 3 · depth 0 / 32K / 100K | 32.2 / 32.9 / 27.3 | 34.1 / 32.3 / 29.2 | +6 / −2 / +7 % |
| 1× Spark · Flash-Next · prefill pp2048 | 1,742 tok/s | 1,768 | +1 % |
| 1× RTX 5090 · 27B NVFP4 · no spec · ~4K | 58.6 tok/s | 55.0 | −6 % |
| 2× Spark TP 2 over ConnectX-7 200GbE · Flash-Next · MTP 4 · 24 req × 2K+512 | 305 tok/s aggregate | 287 | −6 % |
| 2× Spark TP 2 · Flash-Next · MTP 4 · 1 req @ 48K | 47–53 tok/s | 47.5 | −5 % |
| 1× Spark · 27B FP8 · no spec (not in the fit) | 7.94 tok/s | 7.1 | −11 % |
Spark points: Spark Arena llama-benchy runs (pp2048 / tg128, vLLM, NVFP4). RTX 5090 and the 2× Spark points: production traffic and one controlled run on the author’s machines. The FP8 point is from SGLang vs vLLM on a DGX Spark.
Fitted constants
Five free constants against twelve points. Leave-one-out = refit without the point, then predict it.
| constant | value | fitted from | leave-one-out |
|---|---|---|---|
| effective bandwidth fraction | 0.73 | every decode point | −11 … +18 % |
| MFU (dense GEMM) | 0.33 | Spark 27B pp2048 only | −23 % |
| MoE expert-kernel efficiency | 0.71 | Flash-Next decode, 1× and 2× Spark | −11 … +9 % |
| MoE routing cost per token per layer | 5.7 µs | Flash-Next pp2048 only | +55 % |
| per-forward launch overhead | 5.5 ms | intercept of step time vs bytes, decode points | — |
| fixed constant | value | source |
|---|---|---|
| MTP acceptance α per draft position | 0.846 | measured 3.165 tokens/step at k=3 → α + α² + α³ = 2.165; tokens/step = (1 − α^(S+1)) ÷ (1 − α) (Leviathan et al. 2023, eq. 1) |
| all-reduce base latency | 6.6 µs | NCCL baseLat[Ring][LL], tuning.cc v2.27.7 |
| per-step latency NVLink / PCIe / network | 0.6 / 1.0 / 2.7 µs | NCCL hwLat[…][Ring][LL], same file; 5.0 µs per network step with one GPU per box (hwLat[NET][Tree][LL]) |
| per-step latency C2C / Thunderbolt / USB4 / Infinity Fabric | — | no published value: the page asks for one before it simulates a TP group over that link |
| lm_head + embedding bytes | 2 B | assumed (ModelOpt and llm-compressor recipes leave them in BF16) |
| CUDA-graph pool | — | not counted until typed in from a vLLM log |
Limits:
- MFU and the MoE routing cost are each pinned by one prefill point, so their leave-one-out errors (−23 %, +55 %) are the honest prefill error bars. No prefill point exists off the Spark.
- GB10 TFLOPS are measured matmul peaks; every other GPU carries datasheet peaks. MFU fitted on the Spark transfers to datasheet parts with unknown bias.
- The NCCL latencies are the model NCCL uses to pick algorithms, not measurements. Inside one box, vLLM’s custom all-reduce can beat them.
- The 2× Spark residuals are −6 / −4 % at a 10 µs all-reduce, −6 / −5 % at 16.6 µs, −9 / −15 % at 107 µs: this data cannot pin network latency, so the NCCL value is used instead of a fit.
Every constant is editable in the page; the calibration table recomputes with the override.
The model
- Units. GB = 10⁹ B for parameters, traffic and GB/s;
memory_gbis DRAM capacity in GiB (2³⁰ B). - Memory per rank = TP/PP shard of the body at the quant’s bytes per parameter + lm_head and embeddings at 2 B + MTP/extra weights + activations (
max_num_batched_tokens× hidden × 2 B × 8 + fp32 logits); KV pool =memory_gb× 2³⁰ ×gpu_memory_utilization− all of that. KV per token = 2 × full-attention layers × max(kv_heads ÷ TP, 1) × head_dim × kv_bytes; MLA models cache (kv_lora_rank + rope) per layer, replicated over TP. Mixed GPUs: every rank at the per-field minimum. - One forward of T positions is three kernel classes, each
max(bytes ÷ (BW × 0.73), FLOPs ÷ (TFLOPS × 0.33)), added: dense (rest of the body + lm_head), experts (the N·(1 − (1 − k/N)^T) experts actually hit, T × DP under EP, ÷ 0.71) and attention (B × C × KV bytes over every PP stage; 4·T·C·layers·heads·head_dim FLOPs at BF16). - Decode step = forward of ⌈B/PP⌉·(S+1) positions + S draft forwards (one MTP layer + lm_head) + 2·layers all-reduces of 2(TP−1)/TP·hidden·T·2 B over the slowest link in the TP group + (1+S) × 5.5 ms. All-reduce latency = 6.6 µs + one hop per ring step, 2(TP−1) steps. PP: a micro-batch crosses every stage, so idle stages lower the aggregate, not the per-request rate.
- Prefill runs in chunks of
max_num_batched_tokens, each chunk attending to every earlier token; TTFT adds one decode step of the running requests per chunk. - Compare view enumerates every TP·PP·DP = GPUs (EP on/off for MoE) per preset, keeps the ones that fit
max_model_lenand hold one request at the cursor context, and ranks by aggregate tok/s. A row blocked only by a missing link latency shows an inline µs input.
No fallbacks
A null in the catalog (an unstated TFLOPS figure, the B200’s BF16 rate, Llama 3.1’s vocab) blocks the simulation with missing: <field> (reason) until you type a value in the specs editor; the page never substitutes a guess. With Qwen3.8-27B NVFP4 at 32K, 8 of 32 presets compute; the rest are blocked by such nulls or by no FP4 tensor path (Ampere, Hopper). TPU, Gaudi, Intel Arc and Apple parts are listed with their sources but marked not modelled: vLLM reaches them through other backends and the constants were fitted on CUDA kernels. Parts with values derived from rack totals or dev-blog figures carry that in their note.
Sources
The sources panel in the page lists every GPU, box, link, model and quant with its URL and the note on how each number was read. Spark Arena · NVIDIA product pages and the RTX Blackwell whitepaper · HuggingFace config.json files · NCCL tuning.cc · PCI-SIG and IEEE rates · the vLLM docs for gpu_memory_utilization, chunked prefill and expert parallelism.