Qwen3.8 Flash Next,
the Qwen 4 architecture, today.
Qwen’s official preview of the architecture that will underpin Qwen 4 — Gated DeltaNet plus Qwen Sparse Attention, 512 experts. The FP8 weights fit two DGX Sparks wired into one 256 GB pool, served with vLLM behind an OpenAI-compatible /v1 endpoint.
- 180.0B
- 6.0B active
- 262,144 ctx
- FP8 · 2× DGX Spark
- vLLM · /v1
The model from the published weights
- Developer
- Alibaba Qwen
- Released
- August 24, 2026
- Parameters
- 179,999,981,459 (180.0B) total · 6B active per token
- Licence
- Qwen Community 1.0
- Context
- 262,144 tokens native · up to 1,000,000 with YaRN
- Architecture
- 48 layers · 12 × (3 Gated DeltaNet + 1 Qwen Sparse Attention)
- Attention
- QSA · 24 query / 2 KV heads · head dim 256 · block indexer
- Experts
- 512 experts · 10 routed + 1 shared per token
- Draft
- 1 MTP layer · speculative decoding built in
- Input
- text, image, video
Which machine fits tokens of KV cache left
| Weights | Size | DGX Spark · 128 GB | H200 · 141 GB | 2× DGX Spark · 256 GB |
|---|---|---|---|---|
| BF16 | 360.00 GB | does not fit | does not fit | does not fit |
| FP8 | 185.52 GB | does not fit | does not fit | 1,638,4346 × full-context sequences |
Memory vLLM takes (90% of the machine) minus the weights, divided by 27,392 B of BF16 KV cache per token (K + V of 12 QSA layers and 1 MTP layer × 2 KV heads × 256 × 2 B, plus the indexer’s 128-dim keys at 1 per 4 tokens). Each sequence also holds 115,458,048 B of fixed Gated DeltaNet state. One full 262,144-token conversation is one sequence. No official NVFP4 build exists yet.
How it is served one tenant per pool
- Endpoint
- OpenAI-compatible /v1 — chat, completions, streaming.
- Engine
- vLLM, official FP8 weights, tensor-parallel over both Sparks.
- Machine
- Two DGX Sparks wired together as one 256 GB pool.
- Tenancy
- One customer per pool — nobody else’s requests run on it.
- Location
- Poland, EU — on machines we own and run.
- Billing
- Per hour, USD + VAT. Rate set once the service opens.
Run the Qwen 4 architecture on your own pool.
Write what load you expect and how much context you need. Reply within one business day.