knowledge/computer science/ai llm engineering/vllm settings

vllm settings

• 2026-10-24Concurrency (max_num_seqs): how many requests decode at once
• 2026-10-24Context length (max_model_len): the per-request token cap
• 2026-10-24Drafters: draft models, Medusa, EAGLE, MTP, DFlash, DSpark
• 2026-10-24KV offload: a RAM or disk tier behind the KV pool
• 2026-10-24KV pool: kv_cache_memory_bytes vs gpu_memory_utilization
• 2026-10-24KV type: kv_cache_dtype
• 2026-10-24Memory budget: what one start must fit
• 2026-10-24vLLM settings, explained
• 2026-10-24Prefill budget: tokens per engine step
• 2026-10-24Prefix caching (--enable-prefix-caching)
• 2026-10-24Speculative tokens (num_speculative_tokens)