Web scraper token cost per page, measured
One robertz.co article page = 4.7–5.9k tokens through any reader, 17.0k as raw HTML, 1.5k with LLMLingua-2 stacked on Trafilatura. A browser farm’s JSON envelope adds +300–597 tok/page on top of the content — the model pays for the metadata.
Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs
Run locally on my fleet.
- Pages: U1 robertz.co/ (homepage, a listing page) · U2 eight authentication methods battle (long article) · U3 broker_parrot self-hosted job fleet (long post)
fetch: separate run, N=3, cache bypassed.Per URL, per reader
| Reader · config | U1 homepage | U2 auth article | U3 broker post | mean U2+U3 |
|---|---|---|---|---|
| raw HTML | 29,552 | 16,010 | 17,947 | 16,979 |
| Trafilatura 2.0.0 (precision, no links, tables) | ✗ 60 | 5,527 | 3,928 | 4,728 |
| readability-lxml 0.8.4.1 | ✗ 51 | 5,746 | 4,147 | 4,947 |
| Crawl4AI 0.9.4 raw markdown | 1,720 | 6,800 | 4,953 | 5,877 |
| Crawl4AI 0.9.4 fit markdown (Pruning 0.48) | 1,269 | 6,590 | 4,628 | 5,609 |
| Firecrawl onlyMainContent: true | 1,479 | 6,364 | 4,724 | 5,544 |
| Firecrawl onlyMainContent: false | 1,758 | 6,519 | 4,874 | 5,697 |
| Trafilatura → LLMLingua-2 @ rate 0.33 | 60 → 18 | 5,527 → 1,709 | 3,928 → 1,239 | 4,728 → 1,474 |
chromium MCP fetch browser — content ‡ | 405 | 5,527 | 3,986 | 4,757 |
chromium MCP fetch browser — whole envelope ‡ | 705 | 6,076 | 4,583 | 5,330 |
| envelope overhead ‡ | +300 | +549 | +597 | +573 |
| Camoufox rendered HTML § | 31,597 | 16,714 | 19,077 | 17,896 |
| Camoufox → Trafilatura § | 120 | 5,527 | 3,928 | 4,728 |
| nodriver rendered HTML § | 31,597 | 16,750 | 19,113 | 17,932 |
| nodriver → Trafilatura § | 120 | 5,527 | 3,928 | 4,728 |
✗ U1 is a listing page — main-content readers keep almost nothing (Trafilatura kept only the hosting blurb, twice). ‡ separate run, N=3: raw HTML then U1 31,159 tok (homepage grew), U2/U3 unchanged; same-run Trafilatura direct 30 / 5,710 / 4,000, Crawl4AI raw 1,840 / 6,800 / 4,953, fit 1,389 / 6,590 / 4,628. § stealth browser, separate run: Trafilatura on the rendered DOM — same tokens as on fetched HTML for U2/U3. ↔ scroll sideways for every column.
The envelope: what the model pays for besides the page
chromium fetch ‡ | U1 | U2 | U3 |
|---|---|---|---|
| content tok | 405 | 5,527 | 3,986 |
| envelope metadata tok | 300 | 549 | 597 |
| metadata share of what the model reads | 43% | 9% | 13% |
18 metadata keys (status, cache, URLs, timestamps, two SHA-256 hashes, JSON-LD, …) wrap the content → Output formats.
One page, three layers
U2 robertz.co/knowledge/.../eight-authentication-methods-battle/
raw HTML 16,010 tok
→ Trafilatura (precision) 5,527 tok −65% (2.9×)
→ LLMLingua-2 @ rate 0.33 1,709 tok −69% (~9.4× total vs raw HTML; +~8.8 s CPU)Latency per page
| Reader | per page | runs in |
|---|---|---|
| readability-lxml | ~19 ms | pip lib |
| Trafilatura | ~42 ms | pip lib |
| Crawl4AI | ~650 ms | container + Chromium |
| Firecrawl (self-host) | ~940 ms | 5 containers |
chromium MCP fetch, cold ‡ | 5,427–8,293 ms median (4,901–9,026 per call) | browser farm |
chromium MCP fetch, cache hit ‡ | 2–3 ms | shared page cache (7 days) |
| same run ‡: Trafilatura direct | 229–242 ms | pip lib, incl. fetch |
| same run ‡: Crawl4AI raw / fit | 372–467 / 377–410 ms | local MCP service |
| Camoufox § | 748–893 ms | Firefox build + Xvfb |
| nodriver § | 619–835 ms | Chrome + Xvfb |
| + LLMLingua-2 @0.33 | +~8.8 s (U2) · +3.9 s (U3) | CPU |
‡ § N=3 medians, § fresh browser per render. chromium mode=http / mode=auto: 0/18 ok — http path broken at measurement time, not scored.
Budgeting against search
| Item | Tokens | × one search |
|---|---|---|
| one search (10 snippets, previous stage) | 556 | 1× |
| one article page, any reader | 4,728–5,877 | 8.5–10.6× |
one article page, chromium fetch envelope | 5,330 | 9.6× |
| one article page, Trafilatura → LLMLingua-2 | 1,474 | 2.7× |
| one article page, raw HTML | 16,979 | 30.5× |
What it means
- The win is “not HTML”. Every reader cuts raw HTML ~3–3.6× (16,979 → 4.7–5.9k); which reader matters far less.
- On a lean site the filters barely matter. Crawl4AI raw→fit −3…−7% on articles (−26% homepage); Firecrawl main vs raw −2…−3% (−16% homepage).
- Compression stacks. U2: 16,010 → 5,527 → 1,709 (~9.4× total), at +~8.8 s CPU; U3 3,928 → 1,239.
- A stealth browser costs latency, not tokens. Camoufox / nodriver + Trafilatura = 4,728 tok (same as Trafilatura) at 0.6–0.9 s; both passed the Cloudflare page every other reader failed → Extraction quality.
- The envelope is a tax. chromium
fetch= Trafilatura’s text + 300–597 tok of metadata per page (43% of U1, 9–13% of an article). - A browser farm costs ~12–22× Crawl4AI’s latency for the same tokens (5.4–8.3 s vs 0.37–0.47 s, same run); it wins only on a cache hit (2–3 ms) or a page that needs a real logged-in browser.
- Pip readers are ~15–50× faster than browser readers (19–42 ms vs 650–940 ms).
- Prior run, different pages (third-party, boilerplate-heavy: claude.com/pricing, docs.searxng.org, runpod.io): Crawl4AI raw→fit −45…−89% (mean −70%), Firecrawl main vs raw −22…−69%.