Deep-research subagents bottleneck on prefill — how my scraper fixes it

· updated

On a local GPU, a deep-research subagent’s first token waits on prefill — and prefill grows with every token of context it inherits. Measured below on a self-hosted Qwen3.8-27B; the fix is open source: robertziel/scrape-skill.

The before

Two live deep-research lead sessions: 59.2K of a 262.1K window after 9 compressions, and 58.8K of a 65.5K window, 90 percent full Live deep-research leads: context carried into every call desk sandbox 262.1K window 59.2K / 262.1K · 23 % · compressed 9× deep_research tmux 65.5K window 58.8K / 65.5K · 90 % · full 0 25 50 75 100 % share of the context window
Two live leads before the change: 59.2K of 262.1K (9 compressions) and 58.8K of 65.5K; both carry about 59K tokens into every call.

Prefill is the wait

Cold prefill seconds against prompt tokens for seven measured sizes from 1.0K to 87.7K, rising from 0.58 to 39.5 seconds; the 18.1K Hermes floor interpolates to 5.7 s and the 58.5K lead-size point costs 22.4 s Cold prefill time vs prompt size 0 10 20 30 40 cold prefill, s 0 20K 40K 60K 80K prompt tokens 14.7K · 4.49 s 7.4K · 2.29 s 3.7K · 1.31 s 1.0K · 0.58 s 18.1K Hermes floor ≈5.7 s, interpolated 9.63 s 22.4 s 58.5K ≈ a lead’s context 39.5 s Qwen3.8-27B · SGLang · 131K ctx · min of 3 cold calls, unique prompts (no prefix-cache hit), max_tokens=1
Cold prefill (min of 3 unique-prompt calls) at 7 prompt sizes: 0.58 s at 1.0K, 22.4 s at 58.5K, 39.5 s at 87.7K; the 18.1K Hermes floor interpolates to 5.7 s.
Prefill throughput in tokens per second for each measured prompt size: 1,747 at 1.0K, peak 3,270 at 14.7K, falling to 2,218 at 87.7K Prefill throughput by prompt size 0 1,000 2,000 3,000 tokens / s 1.0K 3.7K 7.4K 14.7K 29.3K 58.5K 87.7K prompt tokens 1,747 2,859 3,230 3,270 3,042 2,612 2,218 overhead-dominated -32 % tok/s = prompt tokens ÷ min cold prefill; the 1.0K call is fixed per-call overhead (0.58 s), not GPU speed
Prefill throughput peaks at 3,270 tok/s (14.7K) and falls to 2,218 tok/s at 87.7K (-32 %); the 1.0K point is per-call overhead, not GPU speed.

The prefix cache helps — when it hits

Cold versus warm prefix-cache prefill at 14.7K, 29.3K and 58.5K prompt tokens: 4.49 vs 0.46 s, 9.63 vs 0.63 s, 22.4 vs 0.88 s, speedups 9.8, 15.2 and 25.6 times; one warm 58.5K call missed the cache at 21.0 s and one cold 14.7K call queued behind live traffic at 51.8 s Cold vs warm prefix cache: prefill time per call cold · unique prompt, min of 3 warm · same prompt again 0 5 10 15 20 25 prefill time, s 14.7K 29.3K 58.5K prompt tokens 4.49 s 0.46 s 9.8× 9.63 s 0.63 s 15.2× 22.4 s 0.88 s 25.6× 1 of 2 warm 58.5K calls missed the cache: 21.0 s 1 cold 14.7K call queued behind live traffic: 51.8 s (excluded) warm = best of 2 same-prompt re-sends; cold = min of 3 unique prompts; box shared with live sessions; max_tokens=1
Warm (same prompt re-sent) vs cold prefill: 9.8× at 14.7K, 15.2× at 29.3K, 25.6× at 58.5K; 1 of 2 warm 58.5K calls missed the cache (21.0 s) and 1 cold 14.7K call queued behind live traffic (51.8 s, excluded).

What a Hermes turn actually prefills

Hermes prompt tokens per turn split into fresh prefill and cache hits: the cold first turn is 18,133 fresh plus 64 cached in one call; a lean lead plus one worker is 1,256 fresh plus 36,544 cached across two calls, 96.7 percent cache hits Hermes turns: fresh prefill vs prefix-cache hits cold first turn 1 API call 18,133 fresh + 64 cached lean lead + 1 worker 2 API calls 1,256 fresh 36,544 cached · 96.7 % cache hits 0 10K 20K 30K 40K tok prompt tokens per turn, all API calls fresh = tokens prefilled this turn; cached = prefix served from the KV cache (hermes --usage-file)
Hermes prompt tokens per turn: a cold first turn prefills 18,133 fresh (64 cached); a lean lead + 1 worker (2 calls) prefills 1,256 fresh with 36,544 cached (96.7 % hits).

Inherit vs lean

Derived per-worker cold prefill: a worker inheriting the lead’s 58.5K context costs 22.4 s, a lean worker with about 1.3K fresh tokens on the cached 18.1K floor about 0.6 s, roughly 37 times less Per-worker prefill: inherit the lead’s context vs lean brief (derived) inherit 58.5K ctx, cold 22.4 s lean 1.3K fresh + 18.1K cached ≈0.6 s ≈37× less prefill per worker 0 5 10 15 20 25 s cold prefill per worker call, s derived: inherit = the measured 58.5K point; lean = the 1.0K point (0.58 s) on a prefix-cached floor
Derived from the measured curve: a worker that inherits the lead’s 58.5K context prefills 22.4 s cold; a lean worker (~1.3K fresh on the cached 18.1K floor) ≈ 0.6 s, ≈37× less.
Derived cumulative cold prefill for 1 to 8 workers: inheriting the lead’s context grows by 22.4 s per worker to 179 s at 8, lean workers by 0.6 s each to 4.8 s N workers, cumulative cold prefill (derived) 0 30 60 90 120 150 180 cumulative cold prefill, s 1 2 3 4 5 6 7 8 parallel workers, N 179 s 4.8 s inherit · N × 22.4 s lean · N × 0.6 s derived from the measured curve: prefill is compute-bound on one GPU, so the workers’ prefill adds up
Derived from the measured curve: prefill is compute-bound on one GPU, so N inheriting workers cost N × 22.4 s (179 s at N=8) against N × 0.6 s lean (4.8 s).

How the scraper fixes it

The lead keeps its plan, seen-set and tallies and sends each worker a 401-byte brief; workers on the same model with isolated context fetch through the chromium cache, write records to raw jsonl files and return only a status; each epoch scrapes unseen targets, concludes from the files and re-plans, with raw/seen.txt preventing any page from being fetched or prefilled twice How the scraper keeps prefill flat Lead plan · seen-set · tallies same model as workers window stays flat as data grows lean brief · 401 B recipe · schema · URLs, no transcript worker 1 · fetch → extract → write isolated context · same model worker 2 worker N (≤ max_agents, default 3) {path, n_records, n_failed} status only — records never ride back chromium fetch cache fetched once · reused raw/*.jsonl records land here page bodies stay in the shared page store; a record references them by raw_sha256 epoch loop 1 · scrape unseen N workers in parallel 2 · conclude from raw/ files · tallies 3 · re-plan / pivot highest yield · or stop next epoch · frontier updated raw/seen.txt: no page fetched or prefilled twice
The lead never inherits a worker’s pages: a 401 B brief goes out, records land in raw/*.jsonl, a one-line status comes back, and raw/seen.txt keeps every page to one fetch and one prefill.
/scrape 6        # 6 parallel workers, continuous: epochs until you type `finalise`
/scrape 6 50     # 6 workers, stop after 50 targets

Workers get a 401 B brief and the same model; records go to raw/*.jsonl, pages stay in the fetch cache, and a seen-set stops any epoch re-reading a page. Skill, install notes and MIT licence: github.com/robertziel/scrape-skill.

TL;DR: Cold prefill on a local 27B costs 22.4 s at 58.5K and 39.5 s at 87.7K tokens, and throughput falls 32% as context grows — so subagents that inherit a lead’s ~59K context spend ~22 s each just re-reading it (179 s for 8). A lean worker on a cached 18.1K prefix prefills ~1.3K fresh tokens (≈0.6 s, ~37× less); a measured lead + worker round was 96.7% cache hits. Cache hits aren’t guaranteed (1 of 2 warm 58.5K calls missed: 21.0 s), so the scraper keeps what each worker reads small and writes results to disk instead of back into the lead. → scrape-skill

Questions? Let's Chat :)

open source