Web scraper extraction quality: Trafilatura, readability, Crawl4AI, Firecrawl
Trafilatura 0.84 F1 on the independent WCXB test set (0.96 article-only) — best open extractor. Run locally on ScrapingHub’s 181 pages: Trafilatura 0.955, readability 0.922, Crawl4AI fit_markdown 0.790, Firecrawl main content 0.785 — the browser readers keep the body (recall 0.97) but lose on precision (0.66).
Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs
Independent F1 — published benchmarks
| Reader | WCXB dev F1 | WCXB test F1 | ScrapingHub (article-only) | Speed | Notes |
|---|---|---|---|---|---|
| Trafilatura | 0.79 | 0.84 | 0.958 | fast | pure-Python, no browser |
| Dripper (MinerU-HTML 0.6B) | — | 0.83 | — | 1.57 s/page | small model |
| ReaderLM-v2 (0.5–1.5B) | — | 0.74 | — | 10.4 s/page (slowest) | transformer reader |
| readability | 0.67 | 0.74 | 0.92–0.95 | fast | misses <div> content |
| Crawl4AI / Firecrawl | ✗ no third-party run | ✗ | ✗ | — | published figures are competitor-run / coverage-pass → measured locally, next section |
chromium MCP fetch | = Trafilatura | = Trafilatura | = Trafilatura | 5.4–8.3 s/page cold, 2–3 ms cache hit | runs Trafilatura on the rendered DOM; U2 5,527 tok, identical to Trafilatura |
Third-party benchmarks, not run here. Sources: WCXB arXiv 2605.21097 · ScrapingHub article-extraction-benchmark (github.com/scrapinghub/article-extraction-benchmark).
Same pages, every reader — ScrapingHub benchmark, 181 pages
Run locally on my fleet.
| Reader | Precision | Recall | F1 | Empty pages | Median ms/page |
|---|---|---|---|---|---|
| Trafilatura 2.0.0 | 0.943 | 0.968 | 0.955 | 0 | 10 |
| readability-lxml 0.8.4.1 | 0.913 | 0.931 | 0.922 | 0 | 8 |
Crawl4AI 0.9.4 fit_markdown | 0.664 | 0.976 | 0.790 | 2 | 131 |
Firecrawl (self-host) onlyMainContent:true | 0.660 | 0.967 | 0.785 | 0 | 2,155 |
Firecrawl onlyMainContent:false | 0.490 | 0.985 | 0.655 | 0 | 1,764 |
Crawl4AI raw_markdown | 0.347 | 0.990 | 0.513 | 0 | 131 |
chromium MCP fetch | = Trafilatura | = Trafilatura | = Trafilatura | — | not run |
Scorer: the benchmark's own evaluate.py method (4-word shingles, mean precision and recall over pages, F1 of the means), reimplemented and checked against its published outputs (Trafilatura 0.958 and readability 0.922 reproduced within 0.001). Markdown syntax (links, image refs, #, *, |, emphasis) stripped from every reader's output before tokenizing. Trafilatura: favor_precision, no links, tables, Markdown (published 0.958 = default config). readability: benchmark's own text conversion. Crawl4AI: one /crawl call per page with the HTML passed as raw:, PruningContentFilter 0.48; raw and fit from the same call. Firecrawl: self-host built from source, /v1/scrape from a local static server, removed after the run. ms: in-process for the pip readers, one HTTP round trip for the browser readers.
What each reader kept on robertz.co
Run locally on my fleet.
| Reader | U1 homepage (listing) | U2 auth article | U3 broker post | Body kept? |
|---|---|---|---|---|
| raw HTML | 29,552 | 16,010 | 17,947 | — |
| Trafilatura 2.0.0 | 60 ✗ | 5,527 | 3,928 | articles yes · listing no |
| readability-lxml 0.8.4.1 | 51 ✗ | 5,746 | 4,147 | articles yes · listing no |
| Crawl4AI 0.9.4 fit | 1,269 | 6,590 | 4,628 | yes |
| Firecrawl main content | 1,479 | 6,364 | 4,724 | yes |
Tokens per page. U1 robertz.co/ · U2 /knowledge/computer-science/web-security/eight-authentication-methods-battle/ · U3 /feeds/open-source/broker-parrot-self-hosted-job-fleet/. On U1 Trafilatura kept only the hosting blurb (twice), readability only the name, tagline and email; the browser readers kept the listing.
Where every reader fails
| Prior run: stackoverflow.com behind a Cloudflare JS challenge | Result |
|---|---|
| Trafilatura | fetch blocked |
| readability-lxml | HTTP 403 |
| Crawl4AI | 1 char |
| Firecrawl | HTTP 500 |
Separate run, same page: Camoufox ✓ and nodriver ✓ (real page, accepted answer in the DOM; Trafilatura then kept 631 tok, question + comments, answer dropped); same-run controls Crawl4AI ✗ (blocked: Cloudflare JS challenge), Trafilatura fetch ✗ (nothing downloaded). None of the four ship a stealth browser.
Caveats
- The first F1 table is third-party — WCXB and ScrapingHub published figures on fixed page sets; compare those rows directly.
- The 181-page table is my run — all six on the same ScrapingHub pages and scorer; article pages only, no listing or JS-only pages.
- The robertz.co table is one site — three pages, a spot check, not a ranking.