FlashInfer or Triton for local inference

FlashInfer is a library of finished, hand-tuned CUDA kernels for LLM serving. Triton is a Python-embedded language and compiler that you write GPU kernels in. An inference engine uses both: FlashInfer where a tuned kernel exists, Triton where a format or an operation is new. (OpenAI’s Triton language is unrelated to NVIDIA’s Triton Inference Server.)

The two, side by side

From the FlashInfer and Triton repositories, 2026-10-22.

Where each one runs inside an engine

One served configuration as the example: SGLang 708f51e, Qwen3.8-27B NVFP4 with a DFlash2 speculative drafter, FP8 KV cache, TP=2 across two DGX Sparks (GB10, sm_121).

So “FlashInfer or Triton” is a per-op choice. The flag that matters (--attention-backend in SGLang, VLLM_ATTENTION_BACKEND in vLLM) picks only the attention kernel.

Speed

Published: FlashInfer reports 29–69% lower inter-token latency than compiler-generated backends in LLM serving benchmarks, 28–30% lower latency on long-context inference and 13–17% speedup with parallel generation (paper, datacenter GPUs). No published FlashInfer-vs-Triton attention numbers exist for GB10, and Triton attention speed on GB10 is not measured here.

What the attention backend can win depends on how far the step is from its ceiling. Measured prefill on the example configuration: 60 chunks × 2,048 tokens in 57.6 s (0.96 s per chunk, mean attended depth 46,530 tokens).

Attention is 95 of the 170 µs ideal. Even a perfect attention kernel leaves the measured 469 µs above 374 µs, so swapping the attention backend moves at most ~20% of prefill time here; the rest sits outside the kernel.

Why attention stays in BF16

• Weights are static and quantized offline; K and V are produced per request, at runtime.
• Softmax exponentiates the QKᵀ scores, so a small error in K becomes a large error in the attention weights.
• On sm_121, FlashInfer and SGLang dequantize an FP8 KV cache and compute attention in BF16. The FP8-attention fast path (trtllm_mha) is SM100-only.

So a smaller KV format is a capacity change, not a compute change: more tokens fit in the same memory, and they are dequantized before the math.

Formats: where Triton wins

A new KV format needs a kernel that reads it. In Triton that is a dequant step inside a Python kernel; in FlashInfer it is a new CUDA template per attention variant and architecture. In the same SGLang build, the MX-FP4 KV format is read by Triton and not by FlashInfer:

From fp4_kv_cache_quant_method.py in that build.

Support also depends on the GPU, not only the library:

vLLM rows: read from the images' source, not run. SGLang row: run on 2× DGX Spark.

Speculative decoding: the catch for both

A speculative drafter (DFLASH here) writes and reads a block of draft tokens per step. That is neither plain prefill nor plain decode, so a backend that reads a format “for decode only” or “for prefill only” does not cover it. Every FP4-KV + drafter combination tried on 2× DGX Spark, SGLang 708f51e, 2026-10-22:

With Triton everywhere and a pool sized to free memory, a separate start lost ~28 GB of free memory in ~20 s after “Overriding draft attention backend to triton” and was killed by an 8 GB memory guard. A BF16 dequant workspace sized by the pool’s token count would explain it; the log does not name it.

What dropping the drafter costs, same pair:

The FP8 pool is larger here because the NVFP4 run pinned KV to 28 GB. Pool sizing, checks and the pending same-load speed comparison: KV cache: FP8 vs NVFP4.

Which one to pick

Measured on 2× DGX Spark: FlashInfer with FP8 KV and the drafter is the fastest configuration that serves correctly. Triton reads MX-FP4 KV that FlashInfer cannot, but with a DFLASH drafter it did not serve, so its format advantage does not turn into capacity until the drafter’s write path supports FP4.

ai llm engineering