MXFP4 vs NVFP4 — article mark

MXFP4 vs NVFP4

Same 4-bit number, different scale: MXFP4 spends 0.25 bit/elem on a power-of-two scale per 32 values; NVFP4 spends 0.5 bit/elem on a fractional scale per 16 values. That extra quarter bit is worth +5.4 points on Llama-3.1-8B W4A4 (RTN) and 36 % fewer training tokens at 8B scale.

At a glance

MXFP4NVFP4
OwnerOCP MX spec v1.0 — AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm (Sep 2023)NVIDIA, with Blackwell (2025)
ElementFP4 E2M1FP4 E2M1
Block size3216
Block scaleE8M0 — 8-bit exponent, powers of two only, 2−127…2127E4M3 — FP8 with 3 mantissa bits
Tensor scale—one FP32 per tensor
Bits per element4 + 8/32 = 4.254 + 8/16 = 4.5
Scale rounding errorup to one binade lost on the block maxblock max kept at near-FP8 precision
Who ships weights in itOpenAI gpt-oss MoE layersNVIDIA DeepSeek-R1 / Llama NVFP4 checkpoints
Native tensor-core pathBlackwell, AMD MI350/MI355XBlackwell only

Block anatomy

Block anatomy: one MXFP4 block is 32 E2M1 values plus one 8-bit E8M0 scale, 4.25 bits per element; NVFP4 is 16 E2M1 values plus one 8-bit E4M3 scale per block, plus one FP32 scale per tensor, 4.5 bits per element MXFP4 — 32 × E2M1 + one E8M0 scale = 136 bits E8M0 32 × 4 bit +8 bit, 2n = 4.25 b/elem NVFP4 — 16 × E2M1 + one E4M3 scale, per block = 72 bits · plus one FP32 per tensor E4M3 E4M3 16 × 4 bit +8 bit 16 × 4 bit +8 bit FP32 scale · per tensor keeps every E4M3 block scale in range = 4.5 b/elem (+32 bits per tensor ≈ 0) Dense NVFP4 weights are 5.9 % larger than MXFP4 for the same element count: 4.5 / 4.25.
Block size and scale type are the entire difference. Sources: OCP MX spec v1.0 Table 1; NVIDIA NVFP4 blog.

The element is identical

Both formats store an OCP FP4 E2M1 value: 1 sign, 2 exponent, 1 mantissa, exponent bias 1, no Inf, no NaN. Eight magnitudes, mirrored.

The FP4 E2M1 value grid shared by MXFP4 and NVFP4: plus or minus 0, 0.5, 1, 1.5, 2, 3, 4 and 6, denser near zero, 15 distinct values from 16 codes E2M1 — the same 16 codes under both scales: ±{0, 0.5, 1, 1.5, 2, 3, 4, 6} −6−4−3−202346 0.511.5 −3.4−3.4 → −3 2.42.4 → 2 5.05.0 → 4 · tie, to even step 0.5 below 2, step 1 to 4, step 2 to 6 — every value is a multiple of 0.5, max 6, bias 1, no Inf/NaN
Only the scale changes between formats. Source: OCP MX spec v1.0, Table 5 (FP4 encoding details).

The scale is where the error lives

An E8M0 scale is an exponent only: the block is multiplied by 2n, nothing in between. The OCP spec’s reference conversion sets the scale to the largest power of two ≤ the block max, divided by 4 (E2M1’s largest power of two), then clamps anything above 6. An E4M3 scale has 3 mantissa bits, so the block max can be set to land near 6.

Scale granularity: E8M0 offers only powers of two, so a block whose largest value is 5.0 gets scale 1 and the 5.0 rounds to 4, a 20 percent error; E4M3 offers eight steps per octave, so the scale 0.8125 maps 5.0 to 4.875, a 2.5 percent error Scales available between 0.25 and 2 E8M0 0.250.512 E4M3 ideal 5.0 / 6 = 0.833 log axis · E8M0: 1 scale per octave · E4M3: 8 scales per octave (3 mantissa bits) Worked example — a block whose largest value is 5.0 MXFP4 · E8M0 scale = 2⌊log₂ 5⌋ / 4 = 4 / 4 = 1 5.0 / 1 = 5.0 → not on the grid → rounds to 4 stored value = 4 × 1 = 4.0 error on the block max: −1.0 (−20 %) NVFP4 · E4M3 scale = 5.0 / 6 = 0.833 → E4M3 0.8125 5.0 / 0.8125 = 6.15 → clamps to 6 stored value = 6 × 0.8125 = 4.875 error on the block max: −0.125 (−2.5 %) Our arithmetic on the spec's reference conversion (round-to-nearest-even, clamp above 6). Any block max in (4, 6) or (2, 3) or (1, 1.5) lands off-grid under a power-of-two scale: NVIDIA's own example line measures MSE 0.72 (E8M0) vs 0.08 (E4M3).
Power-of-two scales can waste up to one binade of E2M1's 3.58-binade range on the block maximum. Sources: OCP MX spec v1.0 §6.3; NVIDIA NVFP4 blog, Fig. 4; NVIDIA NVFP4 pretraining paper, App. B.4.

Accuracy — measured by three groups

SetupBaselineMXFP4NVFP4GapSource
Llama-3.1-8B-Instruct, W4A4 RTN, avg of MMLU-CoT · GSM8k · HellaSwag · WinoGrandeFP16 78.9369.3274.73+5.4Egiazarian et al., ICLR 2026
same, W4A4 GPTQ78.9370.6275.72+5.1same
same, W4A4 MR-GPTQ (block-wise Hadamard)78.9373.6575.84+2.2same
Llama-3.1-8B-Instruct, W4A4 direct cast, avg of 6 tasksBF16 70.5361.2567.02+5.8Chhugani et al., 2026
Qwen3-8B, W4A4 direct cast, avg of 6 tasksBF16 73.4765.5071.48+6.0same
8B hybrid Mamba-Transformer pretraining, 1T tokens, loss vs BF16—~2.5 % rel. error~1.5 %MXFP4 needs +36 % tokens to matchNVIDIA, 2025

Every row is weights and activations in 4-bit; weight-only numbers would be closer. The Hadamard row shows most of the MXFP4 gap is the scale, and is recoverable in software at the cost of a rotation.

Average accuracy on Llama-3.1-8B-Instruct in W4A4: FP16 78.93, MXFP4 RTN 69.32, NVFP4 RTN 74.73, MXFP4 GPTQ 70.62, NVFP4 GPTQ 75.72; axis starts at 60 Llama-3.1-8B-Instruct · W4A4 · avg of 4 tasks (axis 60–80, truncated) 6065707580 FP16 78.93 MXFP4 RTN 69.32 NVFP4 RTN 74.73 · +5.4 MXFP4 GPTQ 70.62 NVFP4 GPTQ 75.72 · +5.1 Egiazarian et al., "Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization", ICLR 2026, Table: W4A4 avg.
Same model, same calibration, same element grid: the five-point gap is the scale.

Hardware

GPUFP4 tensor coresMXFP4NVFP4Dense FP4 peakSource
NVIDIA H100 / H200 (Hopper)✗software kernel (Triton, gpt-oss)✗—Blackwell microbenchmarks · vLLM gpt-oss
NVIDIA B200 / GB200 (Blackwell)✓✓ ue8m0 × 32✓ ue4m3 × 169 PFLOPS/GPU (72 ÷ 8, DGX B200)CUTLASS Blackwell docs · DGX B200
NVIDIA GB10 (DGX Spark)✓✓✓“up to 1 PFLOP FP4” — the sparse figure, dense is ~500DGX Spark
NVIDIA RTX 50 (consumer Blackwell)✓✓✓3,352 AI TOPS sparse → ~1,676 dense (RTX 5090)RTX 50 launch · RTX 5090 specs
AMD MI355X (CDNA 4)✓✓not listed10.1 PFLOPS MXFP4AMD MI355X

On Blackwell both formats run through the same tcgen05.mma kind, mxf4nvf4; the instruction takes either a ue8m0 scale per 32 or a ue4m3 scale per 16, so there is no throughput penalty for choosing MXFP4 — only the accuracy one. NVFP4 is NVIDIA-only; MXFP4 is the format that runs on both vendors’ silicon, which is why gpt-oss ships its MoE weights in it (120b: 117B params in 80 GB; 20b: 21B in 16 GB).

When to pick which

TL;DR: MXFP4 and NVFP4 share the E2M1 element. MXFP4 scales 32 values by a power of two (E8M0, 4.25 b/elem); NVFP4 scales 16 values by an FP8 E4M3 fraction plus an FP32 per tensor (4.5 b/elem). The power-of-two scale costs ~5 points in W4A4 on 8B models and 36 % more tokens in 4-bit pretraining. Blackwell runs both at the same speed; AMD and gpt-oss speak only MXFP4.

Sources

Questions? Let's Chat :)

ai llm engineering