Web scraper output formats: what each page reader returns for robertz.co

The same page, robertz.co, through every page reader — the bytes each one hands to the model, unedited.

Series — web scrapers for local LLMs: Overview · Readers compared · Token cost per page · Extraction quality · Output formats · Previous stage: Web search for local LLMs

Summary

Run locally on my fleet. U1 = https://robertz.co/ (a listing page). U2 = the eight-authentication-methods article. ✗ = a main-content reader on a listing page keeps almost nothing. ‡ separate run (raw HTML then 31,159 tok on U1, homepage grew). § separate run, stealth browser render → Trafilatura on the rendered DOM.

1 · Raw HTML — 68,647 bytes, 51 lines, 29,552 tokens

<!DOCTYPE html><html lang="en"> <head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>Robert Zieliński — software engineer · robertz.co</title><meta name="description" content="Robert Zieliński — software engineer behind robertz.co, leometeo.com, and dobrapogoda24. Notes on infra, deploys, Python, and Ruby on Rails."><meta name="author" content="Robert Zieliński"><link rel="canonical" href="https://robertz.co/"><!-- Web fonts, one job each:
         Outfit    — every reading and display role (ties to dobrapogoda24)
         Plex Mono — code, data, dates, counts --><link rel="preconnect" href="https://fonts.googleapis.com"><link r
… (67,947 more bytes)

2 · Trafilatura — Markdown

U1 homepage, complete output (60 tokens):

A dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term.

See the three machines →A dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term.

See the three machines →

U2 article, head:

# Eight authentication methods — battle

Every list of authentication methods puts these eight side by side as if they were eight options for the same slot. They aren’t. One of them isn’t an authentication method at all, two of them are the same thing described at different layers, and the one everyone reaches for by default is the one you can’t take back.

Before the contenders, two questions sort the whole field — and they sort it better than any feature list:

**Does the secret travel on every request?**If yes, you have a*bearer*credential: whoever holds it is you. If no, the holder has to*prove*possession of a key that never leaves them.**Can you take it back?**Some credentials are checked against server state, so revoking is a`DELETE`

. Others verify themselves with pure maths, which is exactly why nothing you do can stop them until they expire.

Those two axes give you a 2×2, and every method on the list lands in one of the four boxes.
… (257 more lines)

3 · readability-lxml — plain text

U1 homepage, complete output (51 tokens):

Robert Zieliński 
I'm here to solve your problems. 
So, what's bothering you? Let's start with HI :)
 Software Engineering 3.0 automation ML AI-native HPC web & Ruby on Rails hello@robertz.co

U2 article, head:

Robert ZielińskiKnowledgeComputer ScienceWeb Security Eight authentication methods — battle — article mark Eight authentication methods — battle 2026-08-10 Every list of authentication methods puts these eight side by side as if they were eight options for the same slot. They aren’t. One of them isn’t an authentication method at all, two of them are the same thing described at different layers, and the one everyone reaches for by default is the one you can’t take back.
Before the contenders, two questions sort the whole field — and they sort it better than any feature list:

Does the secret travel on every request? If yes, you have a bearer credential: whoever holds it is you. If no, the holder has to prove possession of a key that never leaves them.
Can you take it back? Some credentials are checked against server state, so revoking is a DELETE. Others verify themselves with pure maths, which is exactly why nothing you do can stop them until they expire.

Those two axes give you a 2×2, and every method on the list lands in one of the four boxes.

 Authentication methods sorted by whether the secret travels and whether verification needs server state

 the secret travels — bearer
 hold it and you are the user
 the secret stays home — proof of possession
 prove the key without sending it
 verifies
 itself
 no lookup
 needs a
 lookup
 state per call

 JWT · OIDC ID token
… (153 more lines)

4 · Crawl4AI — raw_markdown vs fit_markdown

U1 raw_markdown, head (1,720 tokens):

  * [ knowledge ](https://robertz.co/knowledge/)
  * [ simulators ](https://robertz.co/simulators/)
  * [ feeds ](https://robertz.co/feeds/)
  * [ hardware ](https://robertz.co/hardware/)
  * [ rss ](https://robertz.co/rss.xml)


![Portrait of Robert Zieliński](https://robertz.co/media/portrait-hero.webp)
# Robert Zieliński
I'm here to solve  
your problems.
So, what's bothering you? Let's start with HI :) 
  * Software Engineering 3.0
  * automation
  * ML
  * AI-native
  * HPC
  * web & Ruby on Rails


… (29 more lines)

U1 fit_markdown, head (1,269 tokens) — nav, portrait and hero pruned:

[ Featured · Engineering 3.0 · Team architecture Scaling a non-technical team on agents The agent writes every line; four human seats can each say no, and only two of them read the diff. A product owner writes the ticket, every task auto-deploys its own preview stack behind a wildcard proxy, QA returns findings as prompts rather than patches, and a release tag is the only road to `main`. Read the write-up → ](https://robertz.co/knowledge/computer-science/ai-llm-engineering/scaling-non-tech-vibe-coders-engineering-3-0/ "Engineering 3.0 — the pipeline and the org chart for a team where the agent writes all the code") [ Featured · Agentic AI · Local pipeline Automated news & media pipelines Agentic AI turns a forecast into a finished weather-news reel — script, narration, visuals, final cut — with no human in the production loop. A person only approves it before it publishes, all on a home computer. Read how it's built → ](https://robertz.co/knowledge/computer-science/devops/scripting-automation/content-pipeline-weather-reels/ "Automating a content pipeline: forecast data in, finished weather reels out") [ Featured · Open source · Python broker_parrot — a self-hosted job fleet A workflow engine that makes the database you already run the whole broker — no Redis, no RabbitMQ. It claims jobs with `SKIP LOCKED`, heals crashed workers through expiring leases, and runs DAGs on a durable outbox. Defaults to SQLite, so the smallest setup needs no server at all. Read the write-up → ](https://robertz.co/feeds/open-source/broker-parrot-self-hosted-job-fleet/ "broker_parrot — a self-hosted job fleet on the database you already run") [ Featured · Python · Matplotlib ICON & GFS forecast visualisation leometeo.com → ](https://leometeo.com/model/icon/europe/prec-clouds-temp-pressure "ICON & GFS Model Weather Forecast")
A dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term. 
An isometric H200 SXM GPU module, its finned heatsink standing on the board, above two flat DGX Spark slabs joined by one cable.
  * [ NVIDIA H200 141 GB · 4.8 TB/s $3.50/h net ](https://robertz.co/hardware/#h200)
  * [ NVIDIA DGX Spark GB10 · 128 GB unified $0.50/h net ](https://robertz.co/hardware/#spark)
  * [ 2× DGX Spark 256 GB · 200 Gb/s $0.90/h net ](https://robertz.co/hardware/#cluster)

… (20 more lines)

5 · Firecrawl — JSON envelope

/v2/scrape, formats: ["markdown"], onlyMainContent: true. Envelope trimmed: markdown cut, 10 of 36 metadata keys shown.

{
  "success": true,
  "data": {
    "markdown": "[![Detail of the Engineering 3.0 pipeline diagram: … (5,661 chars)",
    "metadata": {
      "title": "Robert Zieliński — software engineer · robertz.co",
      "description": "Robert Zieliński — software engineer behind robertz.co, leometeo.com, and dobrapogoda24. Notes on infra, deploys, Python, and Ruby on Rails.",
      "language": "en",
      "sourceURL": "https://robertz.co/",
      "url": "https://robertz.co/",
      "statusCode": 200,
      "contentType": "text/html",
      "proxyUsed": "basic",
      "creditsUsed": 1,
      "scrapeId": "01a13741-0143-748a-a282-4e8ffdeef9f8",
      … (26 more keys: ogUrl, og:title, twitter:card, og:image, favicon, …)
    }
  }
}

data.markdown, head (1,479 tokens):

[![Detail of the Engineering 3.0 pipeline diagram: a product owner writes the ticket, the agent — highlighted as the only author — implements it on one branch per ticket, and every task auto-deploys its own proxy-routed preview stack. Below, the review row runs pull request, engineer, final QA and tech lead.](https://robertz.co/images/engineering-3-0-pipeline.webp)\
\
Featured · Engineering 3.0 · Team architecture Scaling a non-technical team on agents The agent writes every line; four human seats can each say no, and only two of them read the diff. A product owner writes the ticket, every task auto-deploys its own preview stack behind a wildcard proxy, QA returns findings as prompts rather than patches, and a release tag is the only road to `main`. Read the write-up →](https://robertz.co/knowledge/computer-science/ai-llm-engineering/scaling-non-tech-vibe-coders-engineering-3-0/ "Engineering 3.0 — the pipeline and the org chart for a team where the agent writes all the code")
[Featured · Agentic AI · Local pipeline Automated news & media pipelines Agentic AI turns a forecast into a finished weather-news reel — script, narration, visuals, final cut — with no human in the production loop. A person only approves it before it publishes, all on a home computer. Read how it's built →](https://robertz.co/knowledge/computer-science/devops/scripting-automation/content-pipeline-weather-reels/ "Automating a content pipeline: forecast data in, finished weather reels out")
[![broker_parrot's Flight Deck panel: three hosts side by side with live CPU, GPU, memory and disk gauges, per-queue worker toggles and an hour of thermal and power history each.](https://robertz.co/images/broker-parrot-flight-deck.webp)\
\
Featured · Open source · Python broker\_parrot — a self-hosted job fleet A workflow engine that makes the database you already run the whole broker — no Redis, no RabbitMQ. It claims jobs with `SKIP LOCKED`, heals crashed workers through expiring leases, and runs DAGs on a durable outbox. Defaults to SQLite, so the smallest setup needs no server at all. Read the write-up →](https://robertz.co/feeds/open-source/broker-parrot-self-hosted-job-fleet/ "broker_parrot — a self-hosted job fleet on the database you already run")
[![ICON & GFS Model weather forecast visualisation](https://dobrapogoda24.pl/uploads/model/icon/europe/prec_clouds_temp_pressure/preview_model.webp)\
\
Featured · Python · Matplotlib ICON & GFS forecast visualisation leometeo.com →](https://leometeo.com/model/icon/europe/prec-clouds-temp-pressure "ICON & GFS Model Weather Forecast")

A dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term.
… (39 more lines)

6 · chromium MCP fetch — JSON envelope

mode=browser, extract=text, max_age=0 (cache bypassed). U1, the complete tool result, unedited:

{
  "status": "ok",
  "cache": "miss",
  "canon_url": "https://robertz.co/",
  "final_url": "https://robertz.co/",
  "http_status": 200,
  "fetched_at": "2026-10-14T06:08:18Z",
  "expires_at": "2026-10-21T06:08:18Z",
  "via": "browser",
  "kind": "other",
  "raw_sha256": "7eb01b68fc7b0241cbe3ae6cb6004a8597e9ec6404491f380d341446886a0256",
  "semantic_sha256": "e488c931024446db51679d21e05b9697bb2a02b9eba8a3bc2e838845e10e3fce",
  "title": "Robert Zieliński — software engineer · robertz.co",
  "content_type": "text/html; charset=utf-8",
  "bytes": 75036,
  "jsonld": [],
  "extractor": "trafilatura",
  "content": "# Robert Zieliński\n\nI'm here to solve\n\nyour problems.\n\nSo, what's bothering you? Let's start with HI :)\n\n- Software Engineering 3.0\n- automation\n- ML\n- AI-native\n- HPC\n- web & Ruby on Rails\n\n[Specs & terms →](https://robertz.co/hardware/)\n\nA dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term.\n\n[See the three machines →](https://robertz.co/hardware/)\n\n[View all · 36 →](https://robertz.co/knowledge/)\n\n-\n[Web scraper extraction quality: Trafilatura, readability, Crawl4AI, Firecrawl](https://robertz.co/knowledge/computer-science/ai-llm-engineering/web-scraper-extraction-quality/) -\n[Web scraper output formats: what each page reader returns for robertz.co](https://robertz.co/knowledge/computer-science/ai-llm-engineering/web-scraper-output-formats/) -\n[Native Claude web search vs self-hosted SearXNG for local LLMs](https://robertz.co/knowledge/computer-science/ai-llm-engineering/native-claude-web-search-vs-self-hosted/) -\n[Web page readers for local LLMs: Trafilatura vs Crawl4AI vs Firecrawl](https://robertz.co/knowledge/computer-science/ai-llm-engineering/page-readers-trafilatura-crawl4ai-firecrawl/) -\n[Web scraper token cost per page, measured](https://robertz.co/knowledge/computer-science/ai-llm-engineering/web-scraper-token-cost-measured/)\n\n[View all · 22 →](https://robertz.co/simulators/)\n\n[View all · 4 →](https://robertz.co/feeds/)",
  "content_len": 1411,
  "truncated": false
}

U2 / U3: +549 / +597 tok of envelope. extractor: trafilatura — the content is Trafilatura run on the rendered DOM (U2 5,527 tok = direct Trafilatura). mode=http / auto: http path broken at measurement time — not scored.

7 · LLMLingua-2 — compressed text

Input: the Trafilatura U1 output above. Rate 0.33, 60 → 18 tokens, complete output:

dedicated machine hour one tenant billed hours No minimum term three machines billed hours No minimum term

8 · Camoufox / nodriver — Trafilatura on the rendered DOM §

Rendered HTML: 75,055 bytes, 31,597 tokens (both browsers, byte-identical on U1). Trafilatura output, U1, 120 tokens, identical from both, complete:

A dedicated machine by the hour — one tenant at a time, billed for the hours you actually use. No minimum term.

See the three machines →- Web scraper extraction quality: Trafilatura, readability, Crawl4AI, Firecrawl
- Web scraper output formats: what each page reader returns for robertz.co
- Native Claude web search vs self-hosted SearXNG for local LLMs
- Web page readers for local LLMs: Trafilatura vs Crawl4AI vs Firecrawl vs a Chromium farm
- Web scraper token cost per page, measured

ai llm engineering