setup
The weight file. Dense models move every parameter per token; MoE models only the active experts (default Llama 3.1 8B, the usual local baseline).
The GGUF suffix: bytes per weight, FP16 = 2.0, Q4_K_M ≈ 0.52. The biggest lever on whether weights fit (default Q4_K_M, what ollama pull ships).
The card: VRAM sets how many layers fit, memory bandwidth how fast they decode (default RTX 4090: 24 GB, 1008 GB/s from the datasheet).
The disk llama.cpp streams from once RAM is full: NVMe Gen4 moves 7 GB/s, about 1/140 of a 4090 (default NVMe Gen4).
llama.cpp maps the GGUF so layers can page in from disk; off, the model refuses to load rather than crawl (default on, llama.cpp's use_mmap).
← swipe the scene sideways to pan →
worked example · the defaults
On load the setup is Llama 3.1 8B at Q4_K_M with 4K context on one RTX 4090 (24 GB, 1008 GB/s), 64 GB RAM, NVMe Gen4. Weights = 8.0B × 0.52 B/p = — GB; KV = 2 × 32 layers × 8 kv-heads × 128 × 2 B = — KB per token, — GB at 4,096 tokens. — of 32 layers fit in VRAM, so decode is bandwidth-bound: 1008 GB/s ÷ 4.16 GB = — tok/s on the Ollama tab. On the vLLM tab the same card keeps — GB for the KV pool, enough for — sequences at 4K, and 16 concurrent users decode about — tok/s in aggregate.formulas · assumptions
weights GB = params × bytes per weight ÷ 1e9
KV per token = 2 × layers × kv_heads × head_dim × 2 B
Ollama tok/s = 1 ÷ Σ over layers (layer weights × active fraction ÷ tier bandwidth)
vLLM max batch = floor((VRAM − weights/TP − overhead − 0.8) × TP ÷ KV per sequence), capped by --max-num-seqs
vLLM aggregate tok/s = (TP × bandwidth ÷ active weights) × lanes × (0.6 + 0.4 × saturation)
- Layers are placed VRAM → RAM → SSD in order, each with its share of the KV cache; weights are spread evenly over layers.
- Runtime overhead 1.1 GB (llama.cpp) or 1.6 GB (vLLM) plus 0.4 GB reserve per box; 4 GB of host RAM kept for the OS; RAM streams at 90 GB/s.
- The KV cache stays 16-bit; the tok/s figures are bandwidth ceilings, with no compute or scheduling overhead.
sources
- Model shapes (layers, kv_heads, head_dim, active parameters): each model's
config.jsonon Hugging Face. - Bytes per weight: llama.cpp's quantization table (file size ÷ parameter count for Q4_K_M ≈ 0.52).
- GPU VRAM and bandwidth: vendor datasheets (NVIDIA, AMD); SSD figures are typical sequential reads per PCIe generation.