Memory budget: what one start must fit
Result, 2026-10-24. DeepSeek-V4-Flash-0731 NVFP4, TP=2 on 2x DGX Spark (GB10): one rank holds 102.0 GiB (117.2 GiB available before the start, 15.2 after). With ~114.5 GiB free at Start, the largest KV pin the start gate passes is 14.8e9 B without the drafter, 9.4e9 B with it (computed below).
What one rank holds
| Term | GiB a rank | Source |
|---|---|---|
| Weights | 79.29 | vLLM “Model loading took 79.29 GiB”, generation 235 (shard headers said 76.68) |
Drafter (DSpark, mtp.*) | 5.06 | shard headers, 10.12 ÷ 2; not measured, the drafter has not loaded yet |
KV pool, --kv-cache-memory 10e9 | 9.31 | 10 × 10⁹ B ÷ 2³⁰; backs 181,777 tokens |
| Overhead | 13.4 | 102.0 − 79.29 − 9.31: CUDA graphs up to batch 48, kernel workspaces, activations at an 8,192-token prefill budget |
| Held, no drafter | 102.0 | 117.2 − 15.2 GiB available, head box, generation 235 |
| Start headroom | 8.0 | the load balancer’s start gate |
Overhead is a remainder, not a measurement of each part. Before this start the row carried 4.3 GiB of overhead copied from another model; the real figure is 3.1x that.
The start gate
Per box, before a launch:
total = weights + drafter (if speculative) + KV pin + overhead
+ prefill-budget term + seat term # both 0 here: 8,192 and 8 seats are inside the calibration
needs = total + 8 GiB
start if needs ≤ MemAvailable on every boxThe load balancer's memory estimate; one formula, every constant read from the model's catalog row.
| KV pin | KV GiB | needs, no drafter | needs, drafter | vs ~114.5 free |
|---|---|---|---|---|
| 8e9 | 7.45 | 108.1 | 113.2 | both start |
| 10e9 | 9.31 | 110.0 | 115.1 | drafter refused |
| 16e9 | 14.90 | 115.6 | 120.7 | both refused |
| 20e9 | 18.63 | 119.3 | 124.4 | both refused |
The 20e9 + drafter row is a start the operator asked for on 2026-10-24 and the gate refused: 124.4 GiB needed, ~114.5 free. The largest pins that pass: 114.5 − 8 − 79.29 − 13.4 = 13.81 GiB = 14.8e9 B without the drafter, 8.75 GiB = 9.4e9 B with it.
Why 8 GiB, twice
Two different 8 GiB checks, at two different moments:
| Start gate | Memory guard | |
|---|---|---|
| Runs | before the launch, in the load balancer | during the load, on each box |
| Reads | the estimate | the box’s MemAvailable, every 2 s |
| Acts | refuses the start | kills the rank after 3 polls in a row under 8 GiB |
| Stops at | — | the head serving; a loaded pair runs below 8 GiB by design |
The headroom is 8 GiB because of 2026-09-30: a prefill budget of 16,384 left the pair at 0.8–4.2 GiB available, the kernel log filled with NVRM NV_ERR_NO_MEMORY, and both boxes hung. The configuration that had served for weeks left 9.1 GiB. 8 sits under the second and well above the first.
Generation 234 shows why the guard is needed after the gate. It started at a 16e9 pin while the row still carried the 4.3 GiB overhead guess: 76.68 + 14.90 + 4.3 + 8 = 103.9 GiB needed against ~117 free, so the gate passed it (computed from the pre-measurement row). The real overhead was 13.4 GiB. During kernel warmup the box fell to 7.9 GiB available and the guard killed the start. The next start, at 10e9, is the one serving.
GB10: one memory for the OS and the GPU
- One pool. CPU and GPU share 128 GB of LPDDR5X (NVIDIA spec); the box reports 121.69 GiB total. There is no separate VRAM to fill: whatever the OS, page cache and other containers hold is not available to the rank.
--gpu-memory-utilizationmisfires here. vLLM takes it as a fraction of the box’s total and subtracts only its own usage; what the OS holds never enters it (vLLMrequest_memory, source-read). The KV pool is therefore pinned in bytes with--kv-cache-memory, and the gate checks the box’s real MemAvailable.- Page cache counts as available but is not free. Right after reading a ~98 GB checkpoint, a box showed 21–24 GiB available but only 1.4–3.5 GiB free; the GPU driver returned
NV_ERR_NO_MEMORYduring warmup and the start died (2026-10-10, a Qwen3.8 start on SGLang).vm.min_free_kbyteswas 44 MB. It happens on some starts and not others. - Idle is not loaded. On the earlier Qwen3.8 pair, ~11.3 GiB was available right after the ranks came up and ~4.8 GiB under a saturating load, steady for minutes. Alarms belong on the loaded figure; the transition between two engines is the gate’s job.
What to change when a start does not fit: KV pool (largest movable term), KV type, drafters (5.06 GiB here), prefill budget and concurrency (charged above the calibrated 8,192 tokens and 16 seats).
Part of vLLM settings, explained.