NVFP4 vs 2:4 sparse: where the other half of the petaflop went — article mark

NVFP4 vs 2:4 sparse: where the other half of the petaflop went

A DGX Spark is sold as 1 petaFLOP. A serving panel on two of them reads 5 % of 323 TFLOP/s. Both numbers are correct, and the distance between them is almost entirely structural — not a misconfiguration, and not something 2:4 sparsity would close.

Every measurement below comes from one instrumented two-node TP=2 deployment serving a 512-expert MoE checkpoint in NVFP4, over 3.16 hours of real traffic.

The ladder, top to bottom

NVIDIA’s headline is FP4 with 2:4 structured sparsity. Dense FP4 is exactly half of it. A real GEMM kernel reaches about two-thirds of that. And a 512-expert MoE reaches a sixth of what the kernel does.

From the 1 petaFLOP headline down to measured serving: 1000 sparse spec, 500 dense spec, 323.2 measured kernel ceiling, 52.8 best prefill, 18.4 median real traffic TFLOP/s per GPU — each rung is measured or specified, never interpolated 1000 FP4 2:4 sparse NVIDIA headline — requires 2:4 pruned weights ~500 FP4 dense spec tensor-core peak for dense weights 323.2 measured, idle box a real NVFP4 CUTLASS GEMM at 4096 cubed — 65 % of dense spec 52.8 best this model hits pure prefill — flat at 16.1 to 16.4 % across every config benched 18.4 median, real traffic 5.7 % of 323.2 — 1.8 % of the headline Same convention on consumer Blackwell: an RTX 5090 is quoted at 3,352 AI TOPS sparse, which is ~1,676 dense.
Two of the four drops are marketing convention; the bottom two are physics. This article is about the bottom two.

The roofline is the whole argument

A GB10 pairs ~323 TFLOP/s of NVFP4 with 273 GB/s of memory bandwidth. Divide them and you get the ridge point: 1184 FLOP per byte. A workload below that line is memory-bound and cannot reach peak compute no matter what the tensor cores can do.

This serving workload measures 112 FLOP/byte mixed, and about 26 in pure decode.

Roofline for a GB10: the sloped memory-bound roof meets the 323.2 TFLOP/s compute roof at 1184 FLOP per byte, while the measured workload sits at 26 to 112 FLOP per byte, far down the slope 10 100 1,000 10,000 100,000 arithmetic intensity — FLOP per byte of memory traffic (log scale) 1 10 100 1000 TFLOP/s per GPU (log) memory-bound roof — 273 GB/s compute roof — 323.2 TFLOP/s ridge — 1184 FLOP/byte decode · 26 FLOP/byte ceiling here is 2.2 % of peak compute real traffic · 112 FLOP/byte ceiling 9.5 % of peak — measured 5.7 % pure prefill · 52.8 past the ridge, still only 16 % of peak 2:4 sparsity raises the flat roof to 1000. At 26 FLOP/byte the workload is 1,400× to the left of where that would matter.
Doubling a ceiling you are using 2 % of buys nothing. The binding constraint is the sloped roof, and 2:4 barely touches it.

What 2:4 actually is

In every group of four weights along the reduction dimension, two are forced to zero. The tensor core skips them — but it has to be told which two, so each kept value carries a 2-bit index. That metadata is why the storage saving is not a half.

The 2:4 mechanism: a group of four dense weights becomes two kept values plus a two-bit index each, which the sparse tensor core uses to skip the pruned lanes Dense — every weight is stored and multiplied w0 w1 w2 w3 4 values · 4 lanes of tensor-core work Pruned to 2:4 — two of every four forced to zero, then retrained to recover w0 0 0 w3 2 values · 2 lanes — the halved FLOP cost Stored on disk and in memory w0 w3 idx 0,3 2 values plus 2 bits of index each — the metadata the saving pays for
Half the multiplies, but not half the bytes: the index that tells the hardware which lanes survived has to travel with the weights.

The byte saving is about a quarter, not a half

NVFP4 already spends 4 bits on the value plus an FP8 block scale every 16 weights — 4.5 bits per weight, or 0.5625 bytes. Going 2:4 halves the values but adds the index, so the realistic landing zone is around 3.25 bits.

Bits per weight: dense NVFP4 spends 4 value bits plus 0.5 scale bits for 4.5 total, while 2:4 sparse NVFP4 spends 2 value bits plus 1 index bit plus about 0.25 scale bits for roughly 3.25 Bits per original weight 4 bits — E2M1 value 0.5 dense NVFP4 4.5 2 bits — values 1 — index 2:4 sparse ~3.25 saves ~28 % Estimate, not a measurement: our arithmetic on the standard 2:4 metadata layout. The FLOP ceiling doubles; the byte bill falls by roughly a quarter.
On a memory-bound box only the right-hand saving is worth anything — and it is a quarter, bought with a pruned and retrained model.

Where a decode step actually spends itself

One decode step on this deployment reads about 34 GB of expert weights per GPU — 512 experts across 48 layers at 0.5625 bytes a weight, halved by the tensor-parallel split. Against that it does a few tenths of a TFLOP of arithmetic.

Per decode step: expert weight reads dominate the byte budget while both hardware ceilings sit far from saturation, bandwidth at about 60 percent and compute at 5.7 percent Bytes read per GPU per decode step MoE expert weights — 34 GB everything else against roughly 0.2 to 0.4 TFLOP of arithmetic — the ratio the roofline reads as 26 FLOP/byte How much of each ceiling that uses memory bandwidth 60 % of 273 GB/s compute 5.7 % of 323.2 TFLOP/s Sparsity doubles the bar that is 5.7 % full. It shaves about a quarter off the bar that is 60 % full. Both rates are the engine’s analytic estimates, not hardware counters — see the note below.
The expert weight read is the bill. Everything else on the page is a rounding error next to it.

Concurrency, not sparsity, is the lever

The expert read is a fixed cost per step, and which experts get touched depends on how many tokens are in flight. One stream wakes about 50 of 512 experts; thirty-two streams wake all of them. The weight bill saturates while the token count keeps climbing — which is why the compute share rises with load and nothing else moves it.

As concurrency rises from 1 to 32 streams the distinct experts touched per step grows from about 50 to all 512, the per-step weight read grows from 3.3 to 34 GB, and the measured compute share rises from 1.7 to 7.3 percent Distinct experts touched per step (of 512) ~50 · 3.3 GB/step ~404 · 27 GB/step 512 · 34 GB 1 stream 16 streams 32 streams top-10 routing of 512 experts: more tokens in a step means more of the table is read, until all of it is Measured compute share, by requests in flight 1 req 1.7 % 5–8 6.2 % 13–16 7.3 % 17–24 6.7 %
Batching amortises a read that would otherwise be paid per token. That is the real lever, and it needs no new checkpoint.

Where 2:4 does pay

The technique is not useless — it is aimed at a regime this workload is not in.

Workload2:4 helps?Why
Training / pre-trainingyes, modestCompute-bound by construction; up to 1.2× measured on GPT-2 pre-training
Vision / encoder inference at large batchyesBig dense GEMMs, genuinely compute-bound, cheap to retrain
INT8 TensorRT inference (non-LLM)yesNVIDIA’s documented workflow, and the one with a speedup track record
LLM prefill / large-batch prompt processingmarginalThe one compute-dense LLM phase — but a Qwen3-4B pruned to verified hardware-valid 2:4 measured ~0 % speedup on TensorRT-LLM
LLM decode (batch 1–32)noMemory-bound; doubles a ceiling you use 2–6 % of, saves ~28 % of bytes at best
MoE, any sizeresearch onlyNo shipping kernel exists anywhere

The fourth row is the one to sit with. That is NVIDIA’s own optimiser, NVIDIA’s own runtime, a Qwen model, weights confirmed as genuine 2:4 — and nothing came out.

Where it is supported in software

Layer2:4 status
NVIDIA siliconyes — sparse MMA since Ampere; FP8 from Hopper; FP4 from Blackwell
cuSPARSELt / torch.sparse semi-structuredyes — primitives present and importable
TensorRT (vision, INT8)yes — the documented success case
ModelOpt pruning → TensorRT-LLMon paper — pruning and recovery exist; see the ~0 % result above
vLLMremoved — PR #36799 deleted the Sparse24 integration and its kernels
LLM Compressorremoved — 2of4 dropped “due to lack of hardware support and user interest”
FlashInfer MoE pathnever existed — no 2:4 or semi-structured grouped GEMM

The producer and the consumer were both withdrawn. Upgrading the engine makes this worse, not better.

Why it cannot run on this cluster

Three independent blockers, any one of them sufficient.

Three independent blockers: the checkpoint carries no sparsity, the engine raises on any sparsity config, and no 2:4 grouped GEMM exists for a 512-expert mixture of experts 1 The checkpoint quant_algo: MIXED_PRECISION, no sparsity key anywhere. Quantised, never pruned. 2 The engine compressed_tensors.py raises on any sparsity config. Support was deleted upstream. 3 The kernel no 2:4 grouped GEMM for 512 experts, in any library. Research papers only. And there is nothing to download: two 200-result model-hub queries for this family returned zero sparse builds. The only published 2:4 LLM family cost pruning plus 13B tokens of knowledge distillation to reach 98.4 % accuracy recovery. No 2:4 MoE checkpoint has ever been published.
Not a missing file — a lane the ecosystem tried, shipped one dense-model family into, and withdrew from.

What worked instead

The same instinct — find the biggest byte term in a step and shrink it — paid better than sparsity would have, with no pruning and no retraining. Profiling put the output head at 27 % of every decode step: a 1.18 GiB bf16 matrix that speculative decoding reads about 5.4 times per step. Storing that one tensor as NVFP4 took it to 0.33 GiB.

Converting one bf16 output head to NVFP4 cut it from 1.18 to 0.33 GiB and raised engine steps per second from 16.9 to 22.0, a 30 percent gain Output head, size on the hot path bf16 1.18 GiB NVFP4 0.33 GiB — read ~5.4× per step Engine steps per second, single stream before 16.9 after 22.0 — up 30 %
One tensor, one format change, no retraining — a larger win than 2:4 offers, in the regime where 2:4 offers nothing.

The sparsity that does pay is already running

Static weight sparsity is the one kind this model does not use. It already runs three dynamic kinds, chosen per token at runtime, and each is worth more on a bandwidth-bound box than a fixed 2-of-4 pattern.

Three dynamic sparsity mechanisms already in the model: top-10 of 512 expert routing, sparse attention capped at a 2048-key budget, and 36 of 48 layers using linear attention 10 of 512 expert routing 98 % of expert weights are skipped for any one token why a 360 GB model runs on two desktop boxes 2,048 keys sparse attention an indexer caps attention at a fixed budget of keys cost stops growing with a 262k-token context 36 of 48 linear-attention layers three quarters of the stack keeps a fixed-size state no KV growth on those layers at all Data-dependent sparsity adapts to the token. A 2-of-4 mask is decided once, at pruning time, and costs accuracy forever.
Three kinds of sparsity already carry this deployment. The fourth kind is the one that would not help.

A note on the numbers

The TFLOP/s and GB/s figures here are the serving engine’s analytic estimates, computed per step from the model’s shapes — not hardware counters. For this particular checkpoint they are inflated: the estimator assumes 48 dense-attention layers where the model actually has 12 sparse-attention layers and 36 linear-attention ones, so it over-counts both the arithmetic and the KV traffic several-fold.

That cuts in the argument’s favour. The true compute share is lower than the 5.7 % quoted, and the true arithmetic intensity is further from the ridge — so the case against 2:4 on this workload is stronger than the measured figures make it look, not weaker. The corrected figures — ≈ 7 FLOP/byte in decode, ~37 of 6,144 CUDA cores multiplying — are in 95 % busy, 1–7 % multiplying: why a sparse MoE leaves GPU compute idle.

When to reach for 2:4

TL;DR: NVIDIA’s petaFLOP is FP4 with 2:4 sparsity; dense is half, a real kernel is a third, and a 512-expert MoE gets a sixth of that. The workload sits at 26–112 FLOP/byte against a 1184 FLOP/byte ridge, so the sparse half of the headline was never reachable — and 2:4 would buy ~28 % of bytes, not 2× of anything. It also cannot run: the checkpoint is dense, the engine deleted sparsity support, no 2:4 MoE kernel exists, and no sparse build of this model family was ever published.

Sources

Questions? Let's Chat :)

ai llm engineering