Web scrapers for local LLMs — article mark

Web scrapers for local LLMs

A page reader turns one URL into text for the model: 16,979 tok of raw HTML → 4.7–5.9k through any reader → 1.5k with LLMLingua-2 on top (robertz.co article, mean of 2 pages).

Run locally on my fleet.

Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs

The field: three families, one stage

The page-reading stage: a URL from search goes through a pip reader, a browser reader or a browser farm with a page cache, optionally a compressor, then into the model — with tokens and latency per robertz.co article page URL → page reader → (compressor) → model (robertz.co article page) URL from search pip readers Trafilatura · readability-lxml 4,728–4,947 tok · 19–42 ms no browser · collapse on listing pages (51–60 tok) browser readers Crawl4AI · Firecrawl (self-host) 5,544–5,877 tok · 650–940 ms Chromium · keep listing pages (1.3–1.8k tok) browser farm + page cache chromium MCP fetch · Trafilatura inside 4,757 + 573 envelope tok · 5.4–6.7 s cache hit 2–3 ms · listing page 405 tok compressor (optional) LLMLingua-2 @0.33 4,728 → 1,474 · +3.9–5.5 s LLM

Raw HTML for scale: 16,979 tok/page. Every reader cuts it ~3–3.6×; reader-to-reader spread is only 24%. Cloudflare JS challenge: all four readers failed; Camoufox and nodriver passed (separate run). Browser farm: separate run, N=3, cache bypassed.

Tokens = mean of U2 and U3 (two long robertz.co articles); LLMLingua-2 mean of U2+U3. Latency per page, single run; browser farm = median of 3 cold renders, envelope = JSON metadata the model also reads.

The headline number per deep-dive

Sub-topicHeadline number (robertz.co)Deep-dive
Readers comparedTrafilatura 4,728 tok, ~42 ms vs browser readers 5.5–5.9k, 650–940 ms; Firecrawl = 5 containers, ~4.3 GBReaders compared →
Token cost per pageraw HTML 16,979 → reader 4.7–5.9k → LLMLingua-2 1,474; one page ≈ 8.5–10.6× a search (556 tok)Token cost per page →
Extraction qualityTrafilatura 0.84 F1 (WCXB, independent), readability 0.74; run locally (ScrapingHub, 181 pages): Trafilatura 0.955, Crawl4AI fit 0.790, Firecrawl main 0.785Extraction quality →
Browser farm (chromium MCP fetch)content 4,757 tok (≈ Trafilatura) + envelope 573 tok; cold 5.4–6.7 s vs Crawl4AI 0.38–0.47 s same run (~12–17×); cache hit 2–3 msReaders compared →
Output formatsMarkdown (Trafilatura, Crawl4AI) · plain text (readability) · JSON envelope + 36 metadata keys (Firecrawl) · JSON envelope + 18 keys around Trafilatura Markdown (chromium fetch) · word list (LLMLingua-2)Output formats →

Practical picks

Page typePickBrowserTokens / page
ArticleTrafilatura✓ no~4.7k
Article, context-boundTrafilatura → LLMLingua-2 @0.33✓ no~1.5k (+3.9–5.5 s CPU)
Listing / JS-renderedCrawl4AI fit_markdown✗ yes~1.3k (U1)
Repeat reads / logged-in pageschromium MCP fetch (browser farm + 7-day cache)✗ yes~4.8k + ~0.6k envelope; 2–3 ms on a hit, 5–9 s cold
Bot-protected / CloudflareCamoufox or nodriver → Trafilatura (passed the Cloudflare page all four readers failed)✗ yes~4.7k (same as Trafilatura), 0.6–0.9 s

Crawl4AI runs as a local MCP for Hermes on my fleet.

TL;DR

TL;DR: On robertz.co the win is “not HTML” (16,979 → 4.7–5.9k tok), not which reader. Trafilatura matches the browser readers on articles at ~42 ms with no browser and the best F1 (0.84 WCXB independent; 0.955 on 181 ScrapingHub pages run locally, vs Crawl4AI fit 0.790, Firecrawl 0.785); use Crawl4AI for listing or JS pages; stack LLMLingua-2 when context is the bottleneck (→ 1.5k). A Chromium browser farm (chromium MCP fetch) gives Trafilatura’s tokens + a 300–597 tok JSON envelope at 5–9 s cold; it wins only on cache hits (2–3 ms) and pages that need a real logged-in browser.

Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs

Questions? Let's Chat :)

open source