--max-model-len 131072, --gpu-memory-utilization 0.9, cursor at 1 request and 32K context. The weights take — per GPU, leaving a KV pool of — (— tokens, — requests at 32K). One decode step takes — ms and yields — tokens, so a single request streams at — tok/s (measured: 20.4). A 32K prompt prefills at — tok/s, TTFT — s on an idle engine. Every number above is recomputed from the model at load; the calibration table at the bottom shows how far it sits from each measurement.Pick the boxes, how many GPUs each holds, and the links inside and between them. Every later step shards over these GPUs and waits on these links.
Load a catalog box or cluster as a starting point; every part keeps its source and stays editable. (Starting setup: 1× DGX Spark, the calibration box.)
The switch between boxes. 'direct cable' means the NICs are cabled back-to-back; a switch caps inter-box bandwidth at its port rate. (Default: direct.)
Add a box, or save this cluster so the compare view ranks it against the catalog presets.
Choose the model and the weight format. This fixes the bytes every GPU must read per token and the KV cache each token leaves behind; step 3 shards them over the GPUs from step 1.
Architecture from the model's config.json: layers, hidden, heads, KV heads, experts, full-attention layers. 'custom' lets you type your own. (Default: Qwen3.8-27B, the calibration model.)
Weight format; bytes per parameter include the block scales (NVFP4 0.5625 B). lm_head and embeddings stay BF16 (constant lmHeadBytes). (Default NVFP4: the calibration runs.)
vLLM --kv-cache-dtype. auto follows the model dtype (2 B per element); fp8 halves the cache and the KV read bytes. (Default auto.)
Uses the GPUs and links from step 1 and the model bytes from step 2. Split the model over the GPUs, then see what each GPU has left for the KV cache and how many requests that holds at the context cursor.
Uses the memory map from step 3 (the KV read grows with context) and the links from step 1 (two all-reduces per layer when TP > 1). The ctx slider and the B key in the toolbar move the cursor; run sweeps B.
No new inputs. The prompt is the context cursor, cut into chunks of max_num_batched_tokens (step 3), each chunk interleaved with the B − 1 requests already decoding (step 4); TTFT is the sum.
The constants in the panel below were fitted on these measurements; the table is the residual of the model as it stands, including any constant you override.