Web page readers for local LLMs: Trafilatura vs Crawl4AI vs Firecrawl vs a Chromium farm
A page reader (scraper) turns a URL into text an LLM can read. Measured on three robertz.co pages: U1 the homepage (a listing), U2 a long article, U3 a long post. Contender 5: chromium MCP fetch — headless Chromium render → Trafilatura → JSON envelope, 7-day shared cache.
Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs
Tokens per article page, by reader and config
fetch from a separate run (N=3, cache bypassed). Run locally on my fleet.The comparison
| Reader · config | U1 homepage | U2 | U3 | mean U2+U3 | config knob | browser | license | deploy weight |
|---|---|---|---|---|---|---|---|---|
| raw HTML | 29,552 | 16,010 | 17,947 | 16,979 | — | — | — | — |
| Trafilatura 2.0.0 | 60 ✗ | 5,527 | 3,928 | 4,728 | precision, no links, keep tables | ✓ no | Apache-2.0 | 1 pip lib, ~42 ms/page |
| readability-lxml 0.8.4.1 | 51 ✗ | 5,746 | 4,147 | 4,947 | — | ✓ no | Apache-2.0 | 1 pip lib, ~19 ms/page |
Crawl4AI 0.9.4 raw_markdown | 1,720 | 6,800 | 4,953 | 5,877 | default raw | ✗ yes | Apache-2.0 | 1 container + Chromium, ~650 ms/page |
Crawl4AI fit_markdown | 1,269 | 6,590 | 4,628 | 5,609 | PruningContentFilter 0.48 | ✗ yes | Apache-2.0 | — |
Firecrawl (self-host) onlyMainContent:true | 1,479 | 6,364 | 4,724 | 5,544 | default | ✗ yes | AGPL-3.0 | 5 containers, ~4.3 GB images, ~7.5 min to healthy, ~940 ms/page |
Firecrawl onlyMainContent:false | 1,758 | 6,519 | 4,874 | 5,697 | full page | ✗ yes | AGPL-3.0 | — |
chromium MCP fetch mode=browser ‡ | 405 (+300) | 5,527 (+549) | 3,986 (+597) | 4,757 (+573) = 5,330 | extract=text, max_age | ✗ yes | — | browser farm + shared cache, 5.4–8.3 s/page cold, 2–3 ms hit |
| Camoufox 0.5.8 → Trafilatura § | 120 | 5,527 | 3,928 | 4,728 | headless="virtual", humanize | ✗ yes | MIT (pip pkg) | 1 pip lib + Firefox build + Xvfb, 0.75–0.89 s/page; passed Cloudflare |
| nodriver 0.50.6 → Trafilatura § | 120 | 5,527 | 3,928 | 4,728 | headless=False (Xvfb) | ✗ yes | AGPL-3.0 | 1 pip lib + Chrome + Xvfb, 0.62–0.84 s/page; passed Cloudflare |
↔ scroll sideways for every column. ✗ on U1 = the reader kept almost nothing (see below). ‡ separate run, N=3 medians; (+n) = JSON envelope tokens the model also reads. § stealth browser, separate run, N=3 medians, Trafilatura on the rendered DOM (rendered HTML 17.9k tok/article). Optional stack: Trafilatura → LLMLingua-2 @0.33 on U2: 16,010 → 5,527 → 1,709 tok (~9.4× total; MIT; +~8.8 s CPU).
Same run: browser farm vs controls
| N=3, median ms · tok | U1 homepage | U2 | U3 |
|---|---|---|---|
chromium fetch mode=browser, cold | 8,293 ms · 705 | 5,427 ms · 6,076 | 6,671 ms · 4,583 |
chromium fetch, cache hit | 2 ms · 705 | 3 ms · 6,076 | 2 ms · 4,583 |
| Trafilatura direct (fetch + extract) | 236 ms · 30 | 229 ms · 5,710 | 242 ms · 4,000 |
Crawl4AI raw_markdown | 372 ms · 1,840 | 442 ms · 6,800 | 467 ms · 4,953 |
Crawl4AI fit_markdown | 410 ms · 1,389 | 396 ms · 6,590 | 377 ms · 4,628 |
chromium fetch mode=http / mode=auto | 0/18 ok | — | — |
chromium tok = whole envelope. Cold medians = ~12–22× Crawl4AI, ~24–35× Trafilatura direct (single calls 4.9–9.0 s). mode=http/auto: http path broken at measurement time — not scored. Raw HTML this run: U1 31,159 tok (homepage grew), U2 16,010, U3 17,947.
The config knob, on a lean site
| knob | U1 homepage | U2 | U3 |
|---|---|---|---|
Crawl4AI raw → fit_markdown | −26% | −3% | −7% |
Firecrawl onlyMainContent false → true | −16% | −2% | −3% |
Prior run (different pages, same day): on boilerplate-heavy third-party pages the same knobs cut −45…−89% (Crawl4AI) / −22…−69% (Firecrawl). The knob matters on bloated pages, not on lean ones.
Listing pages: main-content readers collapse
| U1 homepage | tokens | kept |
|---|---|---|
| Trafilatura | 60 | the hosting blurb, printed twice |
| readability-lxml | 51 | the name, tagline and email |
Firecrawl onlyMainContent | 1,479 | the listing |
Crawl4AI fit_markdown | 1,269 | the listing |
Crawl4AI raw_markdown | 1,720 | the listing |
chromium fetch ‡ | 405 | headline + 5 article links (rendered DOM → Trafilatura); Trafilatura direct same run: 30 |
Main-content readers are for articles, not index pages.
Extraction quality
Trafilatura 0.84 F1 (independent WCXB), readability 0.74; same 181 ScrapingHub pages run locally: Trafilatura 0.955, readability 0.922, Crawl4AI fit 0.790, Firecrawl main content 0.785; all four fail a Cloudflare JS challenge, Camoufox and nodriver pass it → Extraction quality.
Verdict
- Trafilatura: same tokens as the browser readers on articles (4,728 vs 5,544–5,877), ~15–22× faster (42 ms vs 650–940 ms), no browser, Apache-2.0, best F1 (0.84).
- Crawl4AI: when a page needs JS or is a listing (1.3–1.7k on U1 where Trafilatura keeps 60).
- Firecrawl: heaviest — 5 containers, ~4.3 GB, ~7.5 min to healthy, ~940 ms/page, AGPL-3.0.
- Camoufox / nodriver: only for bot-protected pages — Trafilatura’s tokens (4,728) at 0.6–0.9 s; the only two that passed the Cloudflare page.
- readability-lxml: fine on articles (4,947), collapses on listings (51).
- chromium MCP
fetch: Trafilatura’s tokens (U2 5,527 = identical) + 300–597 tok of envelope, at 5.4–8.3 s cold — ~12–22× Crawl4AI for the same text. Wins on a cache hit (2–3 ms) or a page that needs a real logged-in browser.