It isn't SGLang: where the 4× on a DGX Spark actually comes from — article mark

It isn't SGLang: where the 4× on a DGX Spark actually comes from

The engine is not the reason. MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark serves Qwen3.8-27B on one DGX Spark at 38.6–51.5 tok/s on code where vLLM sits near 24.5. Run the same box on vLLM 0.27.1 with the same block-diffusion drafter and it does 31.72 tok/s against its own 7.94 un-drafted — 3.99×. The step change is a drafter that is a drop-in for both engines. What is left over divides into the checkpoint and ten ARM cores.

ControlEngineDrafterWeightstok/sSource
Stock decodevLLM 0.27.1noneFP8, 30.6 GB7.940xBakeer
Same engine, block draftervLLM 0.27.1DFlash2 k=7FP8, 30.6 GB31.720xBakeer
Baseline the repo is measured againstvLLMnoneNVFP4, 24 GB24.5HAOyezi
The repoSGLang devDFlash2 k=8NVFP4, 24 GB38.6HAOyezi
The repo, its own clockSGLang devDFlash2 k=8NVFP4, 24 GB50.9–54.6MiaAI-Lab

Three independent single-Spark benchmark repos, one model. The engine swap is the one variable that never moves a number on its own.

The box sets the rules: 273 GB/s

Memory bandwidth: GB10 273 GB/s against RTX 5070 627, RTX 5090 1792, A100 2039 and H100 3350 GB/s Memory bandwidth, GB/s decode at batch 1 reads every weight once per token — this is the ceiling DGX Spark · GB10 LPDDR5X-9400, 256-bit, unified 128 GB DGX Spark · GB10: 273 GB/s 273 RTX 5070 GDDR7 — same 6,144 CUDA cores as GB10 RTX 5070: 627 GB/s 627 RTX 5090 GDDR7 RTX 5090: 1,792 GB/s 1,792 A100 80 GB SXM HBM2e A100 80 GB SXM: 2,039 GB/s 2,039 H100 SXM5 HBM3 H100 SXM5: 3,350 GB/s 3,350
273 GB/s — a 27B checkpoint at 4-bit takes 88 ms to read once. The RTX 5070 ships the same 6,144 CUDA cores behind 2.3× the bandwidth.
GB10value
CUDA cores6,144 (5th-gen tensor cores)
Memory128 GB LPDDR5X-9400 unified, 256-bit
Bandwidth273 GB/s
NVFP4 dense / 2:4 sparse500 / 1,000 TFLOP/s
GPU TDP140 W
CPU10 × Cortex-X925 @ 3.9 GHz + 10 × A725 @ 2.8 GHz

Decode at batch 1 reads every weight once per token. That makes one number decisive and it is not a FLOP count.

CheckpointWeight bytesOne passCeiling
Qwen/Qwen3.8-27B-FP828.5 GiB = 30.6 GB112.1 ms8.92 tok/s
RadixArk/…-NVFP4-BF16-LMHead~24.0 GB87.9 ms11.38 tok/s

Un-drafted decode is already at 89 % of that ceiling

Bandwidth ceiling 8.92 tokens per second for the FP8 checkpoint versus 7.94 measured, and 11.38 for NVFP4 Un-drafted decode is 89 % of the bandwidth ceiling ceiling = 273 GB/s ÷ weight bytes. One forward pass per token, batch 1. No compute headroom is left to find — only fewer passes per token. 0 3 6 9 12 tok/s FP8 30.6 GB: bandwidth ceiling 8.92 tok/s ceiling 8.92 FP8: measured 7.94 tok/s — vLLM 0.27.1, no drafter, no prefix cache 7.94 89 % of ceiling FP8 30.6 GB NVFP4 24.0 GB: bandwidth ceiling 11.38 tok/s ceiling 11.38 no un-drafted figure published NVFP4 24.0 GB 7.94 of 8.92 The engine is not the bottleneck. There is 11 % of the ceiling left to win, in total. A 3.99× speedup cannot come from scheduling, kernels or a rewrite. It has to come from emitting more than one token per weight pass.
vLLM 0.27.1 with no drafter and no prefix cache: 7.94 tok/s against an 8.92 tok/s bandwidth ceiling — 89 %.

vLLM with no drafter and no prefix cache: 7.94 of a possible 8.92. There is no engine rewrite, kernel backend or scheduler fix worth 300 % here, because there is only 11 % of the ceiling left on the table. The sole remaining lever is fewer weight passes per token.

Every speedup is tokens per weight pass

Accepted tokens per step times the un-drafted rate, against measured throughput: no drafter 7.94, MTP 25.1 ideal versus 17.99 measured, DSpark 22.8 versus 19.12, DFlash2 36.6 versus 31.72 Every speedup is tokens per weight pass — minus the drafting tax one engine, one box, one checkpoint: vLLM 0.27.1 + Qwen3.8-27B-FP8 on one DGX Spark ghost bar = 7.94 × accepted tokens per step · solid = measured · gap = what drafting cost 0 10 20 30 40 tok/s no drafter: 1.000 accepted tok/step × 7.94 = 7.94 tok/s if drafting were free no drafter: measured 7.94 tok/s 7.9 7.9 no drafter 1.000 acc/step 1.00× stock MTP k=3: 3.165 accepted tok/step × 7.94 = 25.13 tok/s if drafting were free MTP k=3: measured 17.99 tok/s 25.1 18.0 −28 % MTP k=3 3.165 acc/step 2.27× stock DSpark k=7: 2.875 accepted tok/step × 7.94 = 22.83 tok/s if drafting were free DSpark k=7: measured 19.12 tok/s 22.8 19.1 −16 % DSpark k=7 2.875 acc/step 2.41× stock DFlash2 k=7: 4.607 accepted tok/step × 7.94 = 36.58 tok/s if drafting were free DFlash2 k=7: measured 31.72 tok/s 36.6 31.7 −13 % DFlash2 k=7 4.607 acc/step 3.99× stock 8.92 = one token per weight pass
Accepted tokens per step set the prize; the drafter's own passes set the price. DFlash2 keeps 87 %, MTP 72 %.

One engine, one box, one checkpoint — the clean control. Accepted tokens per step set the prize; the drafter’s own forward passes set the price.

DrafterAccepted tok/stepFree-drafting idealMeasuredKept× stock
none1.0007.947.94100 %1.00×
MTP k=33.16525.1317.9972 %2.27×
DSpark k=72.87522.8319.1284 %2.41×
DFlash2 k=74.60736.5831.7287 %3.99×

Note DSpark: it accepts fewer tokens per step than MTP (2.875 vs 3.165) and is still faster, because it wastes less drafting them.

Why the block drafter keeps more of its prize

One verify cycle: an autoregressive drafter runs k sequential draft passes before the single target pass; a block-diffusion drafter runs one Where the drafting tax comes from one verify cycle. A draft pass reads the 2.6 GB drafter; the target pass reads 30.6 GB. MTP k=3 — autoregressive drafter draft cost grows with k draft pass 1 of 3 — reads the ~2.6 GB draft weights draft pass 2 of 3 — reads the ~2.6 GB draft weights draft pass 3 of 3 — reads the ~2.6 GB draft weights target verify pass — reads all 30.6 GB, emits 3.165 accepted tokens target pass · 30.6 GB 3 draft passes 3.165 accepted · 17.99 tok/s DFlash2 k=7 — block-diffusion drafter draft cost is flat in k draft pass 1 of 1 — reads the ~2.6 GB draft weights target verify pass — reads all 30.6 GB, emits 4.607 accepted tokens target pass · 30.6 GB 1 draft pass 4.607 accepted · 31.72 tok/s A block-diffusion drafter predicts the whole masked block in one forward pass, keeping several candidates per position. Deeper speculation costs nothing extra to draft — so it keeps 87 % of its accepted tokens where MTP keeps 72 %.
MTP runs k draft passes per verify; DFlash2 runs one, whatever k is.

An autoregressive drafter (EAGLE, MTP) emits k tokens in k sequential forward passes, so draft cost grows with k. A block-diffusion drafter predicts the whole masked block in one pass and keeps several candidates per position, so draft cost is flat in k. Output is unchanged either way — verification is exact.

DFlash, publishedHardwarevs autoregressive
gpt-oss-120b8 × DGX B300up to 15× throughput
Gemma 4 31B (vLLM)1 × B3005.8× Math500 · 5.6× HumanEval
Qwen3 8B (SGLang)1 × B2005.1× Math500 · 4.2× HumanEval
vs EAGLE-3, gpt-oss-120bmatched concurrency2.6× coding · 1.8× writing
vs EAGLE-3, Llama 3.1 8Bmatched concurrency3.0× coding · 2.7× writing

Shipped for SGLang, vLLM and TensorRT-LLM. arXiv:2602.06036.

The depth knob has an interior optimum

Speculative depth has an interior optimum: SGLang MTP steps 2 to 6 peak at 3, and vLLM MTP k peaks at 8 on fresh generation but keeps climbing to 15 on edit-heavy text Deeper speculation is not better — the optimum moves with acceptance left: SGLang MTP step sweep, this box. Right: vLLM MTP k sweep, two workloads. 0 5 10 15 20 SGLang · MTP steps · tok/s MTP steps=2: 12.8 tok/s 12.8 2 MTP steps=3: 17.2 tok/s 17.2 3 MTP steps=4: 16.8 tok/s 16.8 4 MTP steps=5: 16.3 tok/s 16.3 5 MTP steps=6: 15.8 tok/s 15.8 6 speculative steps peak 0 10 20 30 40 vLLM · MTP k · tok/s edit-heavy, k=3: 21.3 tok/s edit-heavy, k=8: 32.2 tok/s edit-heavy, k=15: 39.0 tok/s 39.0 fresh generation, k=3: 17.7 tok/s fresh generation, k=8: 18.64 tok/s fresh generation, k=15: 13.39 tok/s 13.39 edit-heavy fresh 3 8 15 speculative tokens k A draft token that is rejected still cost a pass. Raising k past the acceptance rate buys rejected tokens — unless the text is near-copyable, where acceptance stays near 1.
SGLang MTP peaks at 3 steps (17.2 tok/s) and decays; vLLM MTP peaks at k=8 on fresh text but climbs to k=15 on edit-heavy text.
SweepValuesPeak
SGLang MTP steps (this box)2 → 12.8 · 3 → 17.2 · 4 → 16.8 · 5 → 16.3 · 6 → 15.83
vLLM MTP k, fresh generation3 → 17.70 · 8 → 18.64 · 15 → 13.398
vLLM MTP k, edit-heavy3 → 21.3 · 8 → 32.2 · 15 → 39.015
DSpark block size7 = code peak; 5 = +8 % prose / −16 % codeworkload

A rejected draft token still cost its pass. k has to be set from the acceptance rate of the actual traffic, and near-copyable text (edits, diffs, structured output) accepts almost everything — which is why the same flag sweeps in opposite directions on two workloads.

The residual cross-engine gap is the checkpoint

vLLM with the same drafter reaches 31.72 tokens per second on a 30.6 GB checkpoint; scaled for the 24 GB checkpoint SGLang serves, that predicts 39 to 40, against 38.6 measured The residual cross-engine gap is the checkpoint, not the engine decode is bandwidth-bound: throughput scales with 1 ÷ weight bytes. 30.6 ÷ 24.0 = 1.28×. Code-generation probe, one DGX Spark each, both with the DFlash2 block-diffusion drafter. 0 10 20 30 40 tok/s vLLM 0.27.1 + DFlash2 k=7: 31.72 tok/s — FP8 target, 30.6 GB — the only build whose LM head the drafter accepts under vLLM 31.72 vLLM 0.27.1 + DFlash2 k=7 × 1.28 bytes per token: 40.4 tok/s — 31.72 × (30.6 ÷ 24.0) = 40.4 tok/s predicted on a 24 GB checkpoint 40.4 × 1.28 bytes per token SGLang + DFlash2 k=8: 38.6 tok/s — NVFP4 BF16-LMHead, ~24 GB — measured 38.6 tok/s 38.6 SGLang + DFlash2 k=8 −4.5 % vs predicted Two boxes, two harnesses, different days — but the whole 1.22× SGLang edge is accounted for by the 1.28× in weight bytes. Nothing is left over for the engine.
31.72 tok/s on 30.6 GB predicts 40.4 on 24.0 GB. SGLang measures 38.6 — 4.5 % under, not over.

Decode is bandwidth-bound, so throughput scales with 1 ÷ weight bytes. 30.6 ÷ 24.0 = 1.28×, and the measured SGLang-over-vLLM ratio is 1.22×. Nothing is left over to attribute to the engine; the sign of the residual is negative.

The reason vLLM was stuck on the bigger checkpoint is the interesting part:

Constraint
DFlash2 requirementan unquantized LM head
Under vLLMboth 4-bit Qwen3.8-27B checkpoints fail the pre-flight → FP8, 30.6 GB
The repo’s moveRadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead — NVFP4 body, dense BF16 head
Result4-bit weights and a block drafter, which vLLM could not have at the same time

The repo’s real contribution is a checkpoint export, not an engine.

What SGLang does contribute: knobs, each worth single digits

ChoiceAgainstMeasured delta
--cpuset-cpus 5-9,15-19 (the ten 3.9 GHz X-cores)no pinning+2 to +7 % decode
GDN state bf16cookbook float32+3 %
--mamba-full-memory-ratio 4.21stock 0.9un-clamps concurrency
--mem-fraction-static 0.90 · chunk 81920.85 · 2048TTFT, not decode
extra_buffer_lazy radixextra_bufferpinned by measurement
NVFP4 checkpointFP8 checkpoint~+30 %
NGRAM drafterMTP−30 % · rejected
--speculative-accept-threshold-acc < 11.0worse · rejected
Prefill CUDA graphsoffauto-disabled (GDN ≠ GQA)

Without pinning, the tokenizer and scheduler land on the 2.8 GHz efficiency cores roughly half the time. It is real, and it is 5 % against the drafter’s 300 %.

Where the tuned stack loses

SGLang with DFlash2 against the 24.5 tokens per second vLLM baseline: code 38.6, thinking 32.1, QA 20.9, dialogue 15.9, long-form 14.5, long-context agent 13 to 16 It wins on two workloads out of six SGLang + DFlash2, NVFP4 · bars measured from the zero line; dashed line is the un-tuned vLLM NVFP4 baseline on the same box, 24.5 tok/s. Code generation LRUCache + test Code generation: 38.6 tok/s vs 24.5 baseline — win 1.58× 38.6 1.58× Thinking / math reasoning on Thinking / math: 32.1 tok/s vs 24.5 baseline — win 1.31× 32.1 1.31× Knowledge QA Knowledge QA: 20.9 tok/s vs 24.5 baseline — loss 0.85× 20.9 0.85× Short dialogue the daily workload Short dialogue: 15.9 tok/s vs 24.5 baseline — loss 0.65× 15.9 0.65× Long-form writing Long-form writing: 14.5 tok/s vs 24.5 baseline — loss 0.59× 14.5 0.59× Long-context agent accept rate 0.10–0.16 Long-context agent: 13–16 tok/s vs 24.5 baseline — loss 0.53–0.65× 13–16 0.53–0.65× vLLM 24.5 0 10 20 30 40 tok/s, single stream
Code 1.58× and thinking 1.31× over the vLLM baseline; dialogue 0.65×, long-form 0.59×, long-context agents 0.53–0.65×.
WorkloadvLLM NVFP4SGLang + DFlash2Ratio
Code generation24.538.61.58×
Thinking / math24.532.11.31×
Knowledge QA24.520.90.85×
Short dialogue — the daily workload24.515.90.65×
Long-form writing24.514.50.59×
Long-context agent decode24.513–160.53–0.65×

Two wins out of six. The vLLM column is one scenario-blind baseline, not a per-workload measurement — treat it as a line, not six points.

Acceptance falls from 0.36 to 0.45 on short requests to 0.10 to 0.16 on long reasoning, and decode falls with it from 27 to 33 down to 13 to 16 tokens per second Acceptance decays with context — and takes the speedup with it read out of the SGLang server log, same boot. The drafter is guessing the target’s next tokens; long reasoning traces are less predictable, so fewer drafts survive. 0 12 24 36 tok/s short requests: decode 27–33 tok/s, accept rate 0.36–0.45 short requests: decode floor 27 tok/s, accept rate 0.36–0.45 27–33 accept 0.36–0.45 short requests long reasoning: decode 13–16 tok/s, accept rate 0.10–0.16 long reasoning: decode floor 13 tok/s, accept rate 0.10–0.16 13–16 accept 0.10–0.16 long reasoning (agent traffic) vLLM baseline 24.5 — no speculation The crossover Below ~0.25 accepted tokens per step the drafter costs more than it saves, and the tuned stack falls behind plain vLLM. The lever is lowering k from 8 to 4. For long-context agent traffic this stack is a net loss against the engine it is meant to beat. That is the single most important line in the whole benchmark set.
Accept rate 0.36–0.45 → 27–33 tok/s; 0.10–0.16 → 13–16 tok/s, under the 24.5 tok/s no-speculation baseline.
Context regimeAccept rateDecode
short requests0.36–0.4527–33 tok/s
long reasoning (agent traffic)0.10–0.1613–16 tok/s

Single stream only

Concurrency ladder: aggregate throughput rises 56.6 to 227.6 tokens per second from 1 to 16 streams while per-stream falls 56.6 to 28.2 and time to first token jumps from 280 milliseconds to 4.18 seconds The win is single-stream; the fleet pays for it at 16 SGLang + DFlash2 NVFP4, mem 0.90, one boot, synthetic structural-decode fixture. Two panels, two scales — nothing is overlaid on a shared axis. 0 80 160 240 aggregate tok/s 0 32 64 per-stream tok/s 1 stream: aggregate 56.6 tok/s 56.6 2 streams: aggregate 58.4 tok/s 58.4 4 streams: aggregate 111.6 tok/s 111.6 8 streams: aggregate 184.9 tok/s 184.9 16 streams: aggregate 227.6 tok/s 227.6 1 stream: 56.6 tok/s per stream, TTFT 127 ms 56.6 2 streams: 42.4 tok/s per stream, TTFT 202 ms 42.4 4 streams: 33.4 tok/s per stream, TTFT 224 ms 33.4 8 streams: 30.8 tok/s per stream, TTFT 280 ms 30.8 16 streams: 28.2 tok/s per stream, TTFT 4.18 s 28.2 ×1 127 ms ×2 202 ms ×4 224 ms ×8 280 ms ×16 4.18 s concurrent streams · TTFT below On vLLM the same trade is explicit: 4-bit weights with the free in-checkpoint MTP head overtake the block drafter from two streams up, and lead by 23 % at sixteen.
56.6 → 28.2 tok/s per stream as aggregate climbs 56.6 → 227.6; TTFT holds to ×8 and breaks at ×16.
StreamsTTFTAggregatePer stream
1127 ms56.656.6
2202 ms58.442.4
4224 ms111.633.4
8280 ms184.930.8
164.18 s227.628.2

The same trade is explicit on vLLM: 4-bit weights with the free in-checkpoint MTP head overtake the block drafter from two streams up and lead by 23 % at sixteen. And one k does not serve both — DSpark k=14 is the single-stream champion at 58.5 tok/s on edit-heavy text and gives up 43 % of aggregate throughput at eight streams (119.7 vs 208.7).

Also hardware-bound, not a flag
KV per token~32.8 KB
A full 1M-token sequence≈ 33 GB
KV pool≈ 81 GB (2.48M tokens)
Concurrent 1M requests2, whatever MAX_CONCURRENT_REQUESTS says
Two engines resident at once160 GB demanded of 121.6 GiB → OOM, exit 137, hard reboot at mem 0.95

The configuration, for reference

python3 -m sglang.launch_server --model-path /models   --trust-remote-code --mem-fraction-static 0.90 --attention-backend flashinfer   --chunked-prefill-size 8192 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16   --mamba-full-memory-ratio 4.21 --mamba-radix-cache-strategy extra_buffer   --context-length 262144 --speculative-algorithm DFLASH   --speculative-draft-model-path /draft --speculative-num-draft-tokens 8   --reasoning-parser qwen3 --tool-call-parser qwen3_coder --port 8888
# docker: --cpuset-cpus 5-9,15-19 --shm-size 32g --ipc host --gpus all

Setup and caveats

Hardwareone NVIDIA DGX Spark (GB10, SM121, aarch64), 128 GB unified @ 273 GB/s
ModelQwen3.8-27B dense, hybrid attention (48 GDN + 16 full-attention layers), native 262K context, native VLM
EnginesSGLang dev-qwen38-27b-dflash2 · vLLM 0.23.0 / 0.27.1 aarch64
Measured bythree separate owners, 2026-08-18 → 09-08, three harnesses
Not one experimentthe vLLM-side drafter ladder is a different box, day and harness from the SGLang rows; the cross-engine comparison is normalised arithmetic, not an A/B
Clocks differthe repo’s ndec.py net-decode (50.9–54.6) excludes prefill; the 38.6 figure is a third party’s decode clock; wall-time rows include prefill
Noise floorscode deltas < 15 % are noise; the essay probe is ±1 %; two boots of one image differed 6.5 %
Thermal driftessay 19.5 → 18 tok/s over an hour of benching (power cap)
Rooflinecalculated from nameplate bandwidth; GB10 exposes no memory-traffic counter
Not measuredoutput quality. Every speedup here is output-preserving by construction, but nobody ran an eval

Sources

ShorthandWhat it measuredLink
MiaAI-Labthe repo in question — SGLang + three drafters, every flag measured on-deviceMiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark
HAOyezione Spark, same harness across Ollama / vLLM / SGLangHAOyezi/qwen38-27b-dgx-spark-benchmarks
0xBakeerthe vLLM-side drafter ladder and the acceptance figures0xBakeer/Qwen3.8-27B-FP8-on-a-single-DGX-Spark
DFlashthe drafter itself — block diffusion for speculative decodingarXiv:2602.06036 · NVIDIA developer blog
GB10bandwidth, cores, powerNVIDIA DGX Spark hardware docs

TL;DR: stock decode on a DGX Spark is 7.94 tok/s and 89 % of a 273 GB/s roofline, so the only lever is tokens per weight pass. A block-diffusion drafter accepts 4.607 per step and keeps 87 % of them — 3.99×, and it is a drop-in for vLLM too, where it measures 31.72. The 1.22× SGLang shows on top is 1.28× fewer weight bytes from an NVFP4 checkpoint with a dense BF16 head, which is the only way to run 4-bit weights and a block drafter together. The engine contributes nothing measurable; the ARM core pinning contributes 5 %. On dialogue it is 0.65× plain vLLM, on long-context agents 0.53×, and from two concurrent streams up it loses outright.

Questions? Let's Chat :)

ai llm engineering