95 % busy, 1–7 % multiplying: why a sparse MoE leaves GPU compute idle — article mark

95 % busy, 1–7 % multiplying: why a sparse MoE leaves GPU compute idle

TFLOPs only come from big multiplications. A mixture-of-experts layer splits each batch 512 ways, so each expert’s weights are read from memory and used for a handful of rows — the GPU’s cores spend their time waiting for weights and launching small kernels, not multiplying.

6,144 CUDA cores, three workloads

One square = one core, time-averaged; nvidia-smi reads ~95 % in all three.

Dense benchmark: 5,898 of 6,144 CUDA cores multiplying, 246 waiting on memory Dense benchmark — 5,898 multiplying 96 % · 323 TFLOP/s multiplying 5,898 waiting on memory 246 other 0
Dense NVFP4 kernel: 5,898 multiplying, 246 waiting on memory — 323 TFLOP/s.
Prefill step: at most 430 of 6,144 CUDA cores multiplying, 1,167 waiting on memory, 4,547 other Prefill step — ≤ 430 multiplying ≤ 7 % multiplying 430 waiting on memory 1,167 other 4,547
Prefill step: ≤ 430 multiplying, 1,167 waiting on memory, 4,547 other (kernel start-up, sync between the two machines).
Decode step: about 37 of 6,144 CUDA cores multiplying, 3,686 waiting on memory, 2,421 other Decode step — ~37 multiplying 0.6 % multiplying 37 waiting on memory 3,686 other 2,421
Decode step: ~37 multiplying (the short orange run, top left), 3,686 waiting on memory, 2,421 other.
multiplyingwaiting on memoryothernvidia-smi
Dense benchmark5,8982460~95 %
Prefill step≤ 4301,1674,547~95 %
Decode step~373,6862,421~95 % (91–94 % with one request running)

A core holding a kernel counts as busy while it waits.

Same arithmetic, three shapes

Rows per expert per step = tokens × 10 ÷ 512 (each token goes to 10 of 512 experts).

Three shapes of the same arithmetic: a dense benchmark multiplies 8,192 rows by one matrix; a prefill step multiplies 39 rows by each of 512 expert matrices; a decode step multiplies 1.6 rows by about 405 matrices Rows fed to each weight matrix, one MoE layer Dense benchmark 8,192 rows × 1 matrix Prefill step 39 rows × 512 matrices Decode step 1.6 rows × ~405 matrices orange area = rows multiplied per matrix (not to scale: 8,192 : 39 : 1.6); grey = expert not routed to
8,192 rows × 1 matrix · 39 rows × 512 matrices · 1.6 rows × ~405 matrices.
Steptokensrows per expert
Decode801.6
Prefill~2,00039
Break-even~17,000334
Dense benchmark—8,192

One expert: 5.1 µs to read, 0.024 µs to use

One expert in one layer on one GPU is 1.38 MB — 5.1 µs at 273 GB/s — and one row through it is 4.9 MFLOP.

Per expert, best case: dense benchmark 96 percent multiplying, break-even 50 percent, prefill 10 percent, decode 0.5 percent One expert, best case — multiply time vs 5.1 µs weight read multiplying reading weights Dense benchmark · 8,192 rows 96 % multiplying multiply 124 µs · read 5.1 µs Break-even · 334 rows (~17,000 tokens per step) 50 % multiply 5.1 µs · read 5.1 µs Prefill step · 39 rows 10 % multiply 0.59 µs · read 5.1 µs Decode step · 1.6 rows 0.5 % multiply 0.024 µs · read 5.1 µs Calculated, not profiled: 4.9 MFLOP per row at 323 TFLOP/s; 1.38 MB at 273 GB/s.
Share of time multiplying, best case per expert: 96 % at 8,192 rows, 50 % at 334, 10 % at 39, 0.5 % at 1.6.
value
Expert weights per GPU34.0 GB (67.95 GB of a 105.2 GB checkpoint)
One expert, one layer, one GPU34 GB ÷ 48 ÷ 512 ≈ 1.38 MB
Read at 273 GB/s5.1 µs
FLOPs per row2 × 2560 × 3 × 320 ≈ 4.9 MFLOP

A whole step, as timed

Prefill steps take 71 % of wall time; a step takes ~850 ms with prefill in it or ~180 ms without (two-mode fit R² 0.98).

Whole engine step as timed: prefill step 850 ms is at most 7 percent multiplying; decode step 180 ms is under 1 percent multiplying A whole engine step, as timed multiplying waiting on memory other Prefill step · 850 ms · ~2,000 prefill + 80 decode tokens ≤ 7 % multiply ≤ 61 ms · memory ≤ 165 ms · other ~620–685 ms Decode step · 180 ms · 80 tokens < 1 % multiply ~1 ms · weight reads 99–117 ms · other ≤ 48 ms fixed “Other” is by subtraction, not profiled. Bars drawn at the core-equivalent shares above.
Prefill step 850 ms: multiply ≤ 61 ms, memory ≤ 165 ms. Decode step 180 ms: multiply ~1 ms, weight reads 99–117 ms.
Prefill step, 850 mscount × eachms
Linear-attention and MoE blocks84 × 5.9 ms494
Full-attention blocks12 × 13.8 ms166
All-reduces between the two machines96≈ 101
Remainder≈ 89

7 FLOP per byte against a ridge of 1,184

value
NVFP4 dense kernel peak, measured323 TFLOP/s
Memory bandwidth, nameplate273 GB/s
Roofline ridge323 ÷ 273 ≈ 1,184 FLOP/byte
Decode arithmetic intensity≈ 7 FLOP/byte

The batch cannot simply be made bigger

Leverhavebreak-even needs
Decode tokens per step16 concurrent requests × 5 (1 + 4 speculative) = 80~17,000
Concurrent requests16~3,400
KV-cache memoryfits ~26 sessions at 62k context—
Prefill tokens per step 2048 / 4096 / 8192all ≈ 2,900 tok/s ceiling (0.34 ms per token)—
Prefill tokens per step 16,384hung both machines—

What moves throughput: not prefilling

gen tok/s ≈ 250 × (1 − prefill tok/s ÷ 2,900), R² 0.85 — so a cached prefix is worth ~2×. Working set = KV-cache demand ÷ capacity; past 1.0, cached prefixes are evicted.

Generated tokens per second by working-set band: 160 below 0.8, 128 at 0.8 to 0.95, 113 at 0.95 to 1.05, 60 at 1.05 to 1.2, 49 above 1.2 Generated tok/s by working set working set = KV-cache occupancy × (running + waiting requests) ÷ running 0 50 100 150 working set < 0.8: 160 gen tok/s, 105 min, prefix hit 89 % 160 < 0.8 hit 89 % 105 min working set 0.8–0.95: 128 gen tok/s, 35 min, prefix hit 86 % 128 0.8–0.95 hit 86 % 35 min working set 0.95–1.05: 113 gen tok/s, 11 min, prefix hit 89 % 113 0.95–1.05 hit 89 % 11 min working set 1.05–1.2: 60 gen tok/s, 30 min, prefix hit 64 % 60 1.05–1.2 hit 64 % 30 min working set > 1.2: 49 gen tok/s, 24 min, prefix hit 52 % 49 > 1.2 hit 52 % 24 min past 1.05: prefixes evicted
160 → 49 gen tok/s as the working set passes 1.0 and the prefix hit rate falls from 89 % to 52 %.
Working setminutesprefix hitprefill tok/sgen tok/s
< 0.810589 %1,312160
0.8–0.953586 %1,396128
0.95–1.051189 %1,764113
1.05–1.23064 %2,29460
> 1.22452 %2,19149

vLLM’s own estimator overstates compute 7×

vLLM’s built-in compute-utilisation estimate models all 48 layers as dense full-context attention; this model has 12 layers that attend to 2,048 keys and 36 linear-attention layers.

estimator showedrealphantom
Compute48.8 TFLOP/s≈ 7 TFLOP/s87 % of counted FLOPs
Bandwidth92 %≈ 50 %~53 % of counted bytes

The estimator-based figures in NVFP4 vs 2:4 sparse are the inflated ones; the numbers here replace them.

Setup and caveats

Hardwaretwo NVIDIA DGX Spark (GB10), every layer split across both (tensor parallel), vLLM 0.29
ModelQwen3.8-Flash-Next, NVFP4 (NVIDIA’s 4-bit float format) — 48 layers, 512 experts, top-10 routing, hidden 2560, expert intermediate 640 (320 per GPU)
Measured2026-10-02
Per-expert rowscalculated, not profiled
”Other”by subtraction, not profiled
6,144 coresnameplate; counts are time-averaged equivalents
Memory trafficcalculated; the GB10 chip has no memory-traffic counter
Working-set tablecompared across separate server runs, not a controlled A/B

TL;DR: a decode step gives each expert 1.6 rows — 0.024 µs of multiplying after a 5.1 µs weight read. ~37 of 6,144 cores multiply, nvidia-smi says 95 %. Break-even is ~17,000 tokens per step; the engine has 80. The lever is the prefix cache: 160 vs 49 gen tok/s.

Questions? Let's Chat :)

ai llm engineering