Does MoE on vLLM make sense? Qwen3.8-Flash-Next vs dense Qwen3.8-27B on two DGX Sparks — article mark

Does MoE on vLLM make sense? Qwen3.8-Flash-Next vs dense Qwen3.8-27B on two DGX Sparks

Qwen3.8-Flash-Next computes 6B of its 180B parameters per token and is 98.0 GiB in NVFP4. At the same 32–64K tokens of context per request, it decodes 53.0 tok/s vs 27.6 for dense Qwen3.8-27B at 1 request and 159.7 vs 69.4 at 9–12 requests.

Flash-Next (MoE)Qwen3.8-27B (dense)
Active / stored params6B / 180B27B / 27B
Weights98.0 GiB, 51.98 GiB per rank—
Decode, 1 request, 32–64K context53.0 tok/s27.6 tok/s
Decode, 9–12 requests, 32–64K context159.7 tok/s69.4 tok/s
Faster, by concurrency band6 of 60 of 6
Qwen’s 12 text benchmarkswins 12 of 12—

Specs: Qwen3.8-Flash-Next, Qwen3.8-27B. Checkpoints: hibrid48 and nvidia/Qwen3.8-27B-NVFP4. Speeds: our measurements on 2× DGX Spark, vLLM TP=2.

Stored vs computed

Each token goes to 10 of 512 routed experts plus 1 shared expert. That is 6B parameters of arithmetic out of 180B held in memory. On these Sparks only ~37 of 6,144 CUDA cores multiply in a Flash-Next decode step, so routing saves weight bytes read per token, not compute time.

Parameters stored versus active per token: Qwen3.8-Flash-Next stores 180 billion and computes 6 billion; dense Qwen3.8-27B stores and computes 27 billion Params stored vs active per token, billions (2.39 px per 1B) Qwen3.8-Flash-Next 119B MoE body, idle this token 51B n-gram table 6B active (4.8% of the 125B body, 3.3% of 180B) Qwen3.8-27B · dense 27B stored · 27B active 100%
Solid = computed per token. From the model card: "125B with 6B activated, plus 51B n-gram embedding and 4B MTP"; 512 experts, "10 Routed + 1 Shared".

98.0 GiB resident

Every expert must stay resident, because the next token can be routed to any of them. vLLM’s expert parallelism shards experts across ranks (EP_SIZE = TP × DP). With TP=1 there is one rank, so the whole 98.0 GiB sits on it.

Memory in GiB: Flash-Next checkpoint 98.0 (routed experts 63.3, n-gram table 26.8, rest 7.9); 51.98 per rank when loaded on two DGX Sparks Flash-Next memory, GiB (4 px per GiB) Checkpoint routed experts 63.3 n-gram 26.8 98.0 Loaded, per rank (TP=2) 51.98 Split read from the safetensors headers. Rest = attention 3.6, embeddings + head 1.5, MTP 1.5, vision 0.8, shared expert 0.4.
Checkpoint: 105.24 GB summed over every tensor in hibrid48's shards. Per rank: vLLM's "Model loading took 51.98 GiB" (n-gram table split across the two ranks).

Routed experts are 64.6% of the checkpoint (67.95 GB of 105.24 GB). Per token, each MoE layer uses 10 of 512 of them.

Decode speed at equal context

Both models ran on one pair of DGX Sparks, vLLM TP=2 over ConnectX, with MTP speculative decoding, on production traffic. The load balancer logs a perf window every ~60 s with tokens generated, requests running and KV-cache usage. Context per request = KV tokens in use (usage × pool size) ÷ requests running; shared prefix blocks count once. Decode speed falls as context grows, and the two models saw different contexts: Flash-Next 61–114K tokens per request on average, the 27B 28–75K. So the comparison uses only windows at 32–64K per request, the range both models have in every band. Each figure is Σ tokens ÷ Σ seconds over those windows.

Decode tokens per second at 32 to 64 thousand tokens of context per request, by requests running: Flash-Next MoE 53.0, 70.6, 89.9, 121.7, 159.7, 149.5; dense Qwen3.8-27B 27.6, 33.5, 49.8, 64.1, 69.4, 86.5; the MoE is faster in all 6 bands Decode tok/s by requests running (32–64K tokens of context per request) 0 50 100 150 200 53.0 27.6 159.7 69.4 Flash-Next MoE · 149.5 27B dense · 86.5 1 2 3–4 5–8 9–12 13–16 requests running 17–24 is left out: no Flash-Next window in that band logged KV usage.
Solid: Flash-Next hibrid48 (5.9 h of windows at 32–64K, 2026-09-30 to 10-11). Dashed: nvidia/Qwen3.8-27B-NVFP4 (27 min, 2026-10-02).
Requests runningFlash-Next MoE27B denseMoE ÷ denseMinutes (MoE / dense)
153.027.61.92×32 / 11
270.633.52.11×4 / 1
3–489.949.81.81×6 / 1
5–8121.764.11.90×68 / 7
9–12159.769.42.30×24 / 4
13–16149.586.51.73×219 / 3

Without the context match, the 27B looked faster at 9–12 (89.0 vs 74.8 tok/s). That was context, not the model: in that band Flash-Next spent 139 min at 64–128K per request, decoding 34.5 tok/s, against 159.7 tok/s at 32–64K. The dense side has only 1–11 min per band, all from one day.

Flash-Next ran MTP 4 for 93% of its logged time (MTP 2–6 otherwise) at 6–24 seats; the 27B ran MTP 4 at 16 and 32 seats.

A controlled Flash-Next run (2026-09-30): 100 requests × 512 output tokens at 24 seats, 304.9–305.3 tok/s over 3 reps, 0 errors; prefill on 16 real ~32K-token prompts, 3,073 tok/s.

No published benchmark pairs these two models on 2× DGX Spark under the same settings. The two-node entries on Spark Arena run Flash-Next with MTP and the 27B with DFlash or a draft model.

Qwen’s 12 text benchmarks: +0.8 to +22.3 points

Qwen3.8-Flash-Next minus Qwen3.8-27B, points, on 12 text benchmarks: from plus 22.3 on JobBench to plus 0.8 on SWE-bench Pro; Flash-Next leads on all 12 Flash-Next minus 27B dense, points (15 px per point; Qwen's table) JobBench +22.3 55.7 vs 33.4 DeepSWE 1.1 +16.5 58.7 vs 42.2 SWE-bench Multilingual +7.2 81.0 vs 73.8 Toolathlon Verified +6.4 73.5 vs 67.1 NL2Repo-Bench +5.8 48.1 vs 42.3 HLE +5.1 35.9 vs 30.8 Agents' Last Exam pass@1 +3.9 24.3 vs 20.4 CoWorkBench +3.2 73.9 vs 70.7 GPQA Diamond +2.5 91.7 vs 89.2 IFBench +1.8 81.3 vs 79.5 LiveCodeBench v6 +1.6 91.9 vs 90.3 SWE-bench Pro +0.8 62.5 vs 61.7 Qwen's reported scores for the original weights, not our NVFP4 serves.
All 12 text benchmarks in the Qwen3.8-Flash-Next model card, sorted by gap.

TL;DR

TL;DR: At the same 32–64K tokens of context per request on 2× DGX Spark, Flash-Next decodes 1.7–2.3× faster than dense Qwen3.8-27B in all 6 concurrency bands with data, and scores higher on all 12 of Qwen’s text benchmarks.

Questions? Let's Chat :)

hardware