BF16 vs FP16

Same 2 bytes, different split. BF16 spends 8 bits on the exponent and gets FP32’s whole range. FP16 spends 10 on the mantissa and gets 8x finer steps, but only up to 65,504. For a modern LLM, range wins: take BF16.

At a glance

BF16FP16
Bits: sign · exponent · mantissa1 · 8 · 71 · 5 · 10
Exponent bias127 (same as FP32)15
Largest finite≈ 3.39 × 103865,504
Smallest normal2−126 ≈ 1.18 × 10−382−14 ≈ 6.10 × 10−5
Smallest subnormal2−133 ≈ 9.2 × 10−412−24 ≈ 5.96 × 10−8
Steps per power of two1281024
Step next to 1.02−7 = 0.00781252−10 ≈ 0.000977
Worst rounding error, half a step2−8 ≈ 0.39 %2−11 ≈ 0.049 %
From FP32drop the low 16 bits (or round them)re-encode; can overflow or flush to 0
Trainingno loss scalingloss scaling
Tensor-core hardwareNVIDIA Ampere+, AMD CDNA1+ / RDNA2+, TPUevery NVIDIA GPU since Volta, every AMD matrix core
PyTorch dtypetorch.bfloat16torch.float16

FP16 is IEEE 754 binary16. BF16 layout: Google's BFloat16 post (Wang, Kanwar 2019). Limits and steps: ml_dtypes 0.6.0 finfo in python:3.12-slim; Cloud TPU flushes BF16 subnormals to zero, so there the floor is the smallest normal. Rounding error as a share of the power of two below the value.

Bit layouts drawn to scale, one cell per bit. FP32 is 1 sign, 8 exponent and 23 mantissa bits. BF16 is 1 sign, 8 exponent and 7 mantissa bits: exactly the top 16 bits of FP32, so the same exponent and range, max about 3.39 times 10 to the 38, with 128 steps per power of two. FP16 is 1 sign, 5 exponent and 10 mantissa bits: max 65,504, 1024 steps per power of two. Same 16 bits, split two ways (one cell = one bit) FP32 SEEEEEEEEMMMMMMMMMMMMMMMMMMMMMMM 8 exponent · 23 mantissa · max ≈ 3.40 × 1038 BF16 cut: FP32's top 16 bits BF16 SEEEEEEEEMMMMMMM 8 exponent · 7 mantissa max ≈ 3.39 × 1038 · 128 steps / binade FP16 SEEEEEMMMMMMMMMM 5 exponent · 10 mantissa max 65,504 · 1024 steps / binade S sign · E exponent = range · M mantissa = precision
Layouts from IEEE 754 (binary16, binary32) and Google's BFloat16 post. The BF16 row is aligned under FP32: its 16 bits are FP32's top 16.
Bit layouts drawn to scale, one cell per bit, FP32 split over two rows of 16. FP32 is 1 sign, 8 exponent and 23 mantissa bits. BF16 is 1 sign, 8 exponent and 7 mantissa bits: exactly FP32's top row of 16 bits, so the same exponent and range, max about 3.39 times 10 to the 38, with 128 steps per power of two. FP16 is 1 sign, 5 exponent and 10 mantissa bits: max 65,504, 1024 steps per power of two. Same 16 bits, split two ways one cell = one bit FP32 8 exponent · 23 mantissa · max ≈ 3.40 × 1038 SEEEEEEEEMMMMMMM ↑ BF16 cut: FP32's top 16 bits MMMMMMMMMMMMMMMM BF16 8 exponent · 7 mantissa SEEEEEEEEMMMMMMM max ≈ 3.39 × 1038 · 128 steps / binade FP16 5 exponent · 10 mantissa SEEEEEMMMMMMMMMM max 65,504 · 1024 steps / binade S sign · E exponent = range M mantissa = precision
Layouts from IEEE 754 (binary16, binary32) and Google's BFloat16 post. The BF16 row is aligned under FP32: its 16 bits are FP32's top 16.

BF16 is FP32 with the bottom half cut off

A normal value is (−1)S × 1.M × 2E − bias. BF16 keeps FP32’s sign, its 8 exponent bits and bias 127, and the first 7 of its 23 mantissa bits.

Worked example

ValueBF16ErrorFP16Error
3.33.296875−0.09 %3.30078125+0.02 %
0.10.10009765625+0.10 %0.0999755859375−0.02 %
70,00070,144+0.21 %inf—
10−81.0012 × 10−8+0.12 %0−100 %

ml_dtypes 0.6.0 and NumPy 2.5.4 in python:3.12-slim, cast from FP32, round to nearest even.

3.3 = 1.65 × 21. Between 2 and 4 BF16 steps by 2−6 = 0.015625, so the nearest neighbours are 3.28125 and 3.296875: 3.296875. FP16 steps by 2−9 = 0.001953125 and lands on 3.30078125.

Top: representable range on a log axis from 10 to the minus 46 to 10 to the 40. FP32 spans 1.4e-45 to 3.40e38 and BF16 spans 9.2e-41 to 3.39e38, both with normals from 1.18e-38. FP16 spans 5.96e-8 to 65,504, normals from 6.10e-5. A value of 70,000 becomes inf in FP16 and 70,144 in BF16. Bottom: every value from 3.28125 to 3.3125. BF16 has 3, spaced 0.015625; FP16 has 17, spaced 0.001953125. 3.3 rounds to 3.296875 in BF16, minus 0.09 percent, and to 3.30078125 in FP16, plus 0.02 percent. Range: what each format can hold (log scale) FP32 BF16 FP16 subnormal from 1.4 × 10−45 · normal from 1.18 × 10−38 3.40 × 1038 subnormal from 9.2 × 10−41 · normal from 1.18 × 10−38 3.39 × 1038 subnormal from 5.96 × 10−8 · normal from 6.10 × 10−5 10−4010−3010−2010−10100101010201030 65,504 · 70,000 → inf tinted: normal · grey: subnormal (fewer bits of precision) · line: FP16's ceiling · the same 70,000 in BF16 → 70,144 Precision: every value from 3.28125 to 3.3125, and where 3.3 lands 3.3 BF16 → 3.296875 · −0.09 % 3.281253.2968753.3125 FP16 → 3.30078125 · +0.02 % 3.281253.2968753.3125 Step between 2 and 4: BF16 2−6 = 0.015625 · FP16 2−9 = 0.001953125. Next to 1.0: 0.0078125 vs 0.0009765625.
Range bars: smallest subnormal, smallest normal and largest finite value of each format, from ml_dtypes 0.6.0 finfo (Cloud TPU flushes BF16 subnormals to zero). Grid points: every BF16 and FP16 value in the window; rounding is to nearest.
Top panel: representable range on a log axis from 10 to the minus 46 to 10 to the 40. FP32 spans 1.4e-45 to 3.40e38 and BF16 spans 9.2e-41 to 3.39e38, both with normals from 1.18e-38. FP16 spans 5.96e-8 to 65,504, normals from 6.10e-5. A value of 70,000 becomes inf in FP16 and 70,144 in BF16. Bottom panel: every value from 3.28125 to 3.3125. BF16 has 3, spaced 0.015625; FP16 has 17, spaced 0.001953125. 3.3 rounds to 3.296875 in BF16, minus 0.09 percent, and to 3.30078125 in FP16, plus 0.02 percent. Range (log scale) FP32 1.4 × 10−45 to 3.40 × 1038 BF16 9.2 × 10−41 to 3.39 × 1038 FP16 5.96 × 10−8 to 65,504 10−4010−201001020 70,000 → inf tinted: normal · grey: subnormal (fewer bits) normal from 1.18 × 10−38 (FP32, BF16), 6.10 × 10−5 (FP16) line: FP16's ceiling, 65,504 · BF16: 70,000 → 70,144 Precision near 3.3 every value from 3.28125 to 3.3125 3.3 BF16 → 3.296875 · −0.09 % 3.281253.2968753.3125 FP16 → 3.30078125 · +0.02 % 3.281253.2968753.3125 Step between 2 and 4: BF16 2−6 = 0.015625, FP16 2−9 = 0.001953125. Next to 1.0: 0.0078125 vs 0.0009765625.
Range bars: smallest subnormal, smallest normal and largest finite value of each format, from ml_dtypes 0.6.0 finfo (Cloud TPU flushes BF16 subnormals to zero). Grid points: every BF16 and FP16 value in the window; rounding is to nearest.

Training: FP16 needs loss scaling, BF16 does not

Small gradients fall below FP16’s range and become 0. NVIDIA’s mixed-precision guide: “gradients may have to be scaled to move them into the range to keep them from becoming zeros in FP16”. So FP16 training multiplies the loss by a scale before the backward pass and divides the gradients after, and the scale is adjusted when an overflow shows up.

BF16 reaches down to 1.18 × 10−38 like FP32, so there is nothing to shift. Google’s TPU team: “Unlike FP16, which typically requires special handling via techniques such as loss scaling, BF16 comes close to being a drop-in replacement for FP32 when training and running deep neural networks”.

Inference: use the checkpoint’s BF16

For inference on a modern GPU: BF16, or auto on a BF16 checkpoint. The 16-bit KV cache is BF16 as well, 2 bytes per element. That is the baseline FP8 E4M3 vs E5M2 weighs E4M3 against: half the bytes, 16 steps between 1 and 4 instead of 256.

Every kv_cache_dtype and its bytes: KV type. The 4-bit end: KV cache: FP8 vs NVFP4.

Sources

Questions? Let's Chat :)

ai llm engineering