How SERP works

A SERP API is a browser that reads search pages for you and hands back the rows. SERP means search engine results page: the page Google or Bing shows after you type a query. Google, Bing and the rest do not offer their results as a free JSON feed, so a SERP API loads that page the way a person would, finds the result blocks in the HTML, and turns each into {rank, title, url, snippet}.

This article walks one request through OpenSERP, an MIT-licensed Go server that does this for Google, Bing, DuckDuckGo, Yandex, Baidu and Ecosia. All numbers below were measured on 2026-10-28: OpenSERP at commit b6f9637 (v0.8.15 + 3 commits) built from source, run in Docker on one home connection, no proxy.

The path of one request

GET /google/search?text=how+does+tcp+slow+start+work&limit=10

1. Cache

Identical requests inside 120 seconds are answered from memory. The key is the engine, the query and every parameter (language, region, limit, page, filters). The second run of each query above returned in under 1 ms with took_ms: 0. Only non-empty JSON responses are cached, and a response served by a fallback engine is not.

2. Rate limiter and circuit breaker

Each engine has its own token bucket: 60 requests per minute, 3 allowed back to back. A circuit breaker per engine opens after 5 failures: for the next 60 seconds that engine fails immediately instead of holding a Chrome tab for up to 15 seconds per attempt, then 2 successful test requests close it again. Timeouts and network errors are retried once; parser failures are not. A captcha, block or 429 is not retried through the same IP; with a proxy pool of 2+ healthy proxies, it is retried once through the next proxy and the challenged one is moved to the back.

3. The engine URL

The query becomes the URL a person’s browser would send. For Google:

The example request sends only the first two rows to www.google.com. Every engine has its own builder; the parameter names differ, the idea does not.

4. Load the page in Chrome

The default mode drives headless Chromium through the Chrome DevTools Protocol (Go library go-rod). Per search:

• A fresh browser context. Like a new incognito window: own cookies, own storage, optionally its own proxy, disposed after the page is read. With a proxy session ID, a “lane” keeps that session’s cookies between searches instead. Up to 6 Chrome processes run at once; an idle one is closed after 5 minutes.
• A coherent browser profile. 13 built-in profiles, each a consistent set of user agent, platform, client-hint headers, screen size, time zone and languages. The Chrome version in the user agent is rewritten to the version actually running, so the header and the JavaScript-visible values agree.
• Heavy content blocked. Images, fonts, CSS and media are not downloaded (except behind an authenticated proxy), and known tracker domains are refused. The result rows are in the HTML; the pictures are not needed to read them.
• Wait for the result blocks. Google is read once div.tF2Cxc (an organic result wrapper) appears. If it never does, a broader selector on Google’s data-hveid/data-ved attributes is tried, keeping only blocks that carry data-ved.

5. Classify the page

A search engine that suspects a bot often does not answer with an error code. It answers 200 with a different page. So before parsing, the page is sorted:

Without this step a captcha page parses as “0 results”, and the caller believes the web has nothing on the topic. A captcha solver (2captcha) can be switched on for Google; it is off by default.

6. Parse, unwrap, normalise

Each result block yields a title, a link and a snippet. Sponsored blocks carry their own markers; they come back with "type": "ad" instead of "organic", not dropped. Two things then have to be fixed.

Links are wrapped. Search engines route clicks through themselves to count them:

When a Google /url or /goto link carries no readable destination, OpenSERP follows the redirect: 4 at a time, inside a 3-second budget per search. After a challenge on one exit IP, resolving through that IP pauses for 10 minutes.

Rows are normalised. Each row gets a stable ID, its absolute position across pages, the domain and a domain category. Rank 1 of the /mega/search response further down (from DuckDuckGo):

{
  "id": "s_6c5765c2969eaedd",
  "rank": 1,
  "type": "organic",
  "title": "TCP slow start - Glossary | MDN",
  "url": "https://developer.mozilla.org/en-US/docs/Glossary/TCP_slow_start",
  "display_url": "developer.mozilla.org › en-US › docs › Glossary › T…",
  "snippet": "Jul 16, 2025TCP slow start is an algorithm that helps discover the available network bandwidth …",
  "domain": "developer.mozilla.org",
  "position": { "absolute": 1 },
  "engine": "duckduckgo",
  "domain_info": { "tld": "org", "sld": "mozilla", "category": "" }
}

"Jul 16, 2025TCP" is verbatim: the page’s date and the snippet text arrive in one element, and the parser joins them without a space. Every SERP parser is a list of selectors written against markup the engine can change at any time; when Google renames a class, rows shrink or the request fails with 504 search_timeout. Watch row counts as well as status codes.

Why a browser at all: raw HTTP, measured

Parsing HTML does not need Chrome. OpenSERP also has a raw mode: one HTTP request per search, sent through tls-client, a Go HTTP client whose TLS handshake imitates Chrome’s, then parsed with goquery. No browser process, no tab, no JavaScript. Same four queries, both modes:

Times are the mean took_ms over 4 different queries, limit=10, cold cache.

OpenSERP classified Google’s raw-mode answer as 403 blocked on all four queries, the first after 163 ms. A TLS handshake that looks like Chrome was not enough. That is why browser mode is the default and the only mode every engine supports; raw mode is the faster option for engines that tolerate it.

The row counts also move. Bing returned 10, 3, 1 and 10 rows for four queries with limit=10, all HTTP 200. A SERP API returns what the engine showed that request, and limit is a ceiling, not a promise.

Several engines at once

/mega/search sends the query to several engines in parallel and merges them. For the TCP query across Google, Bing, DuckDuckGo and Yandex:

Duplicates are matched on a normalised URL. Each unique URL becomes a cluster with a score:

score = sum(1 / rank for each engine that returned it) / engines queried

MDN’s “TCP slow start” page was rank 1 on DuckDuckGo and rank 4 on Yandex: (1/1 + 1/4) / 4 = 0.31, the top cluster. Only 3 of the 34 URLs came back from more than one engine, which is the argument for asking more than one. A slow engine does not hold the response forever: after 90 seconds the merge returns with whatever arrived and lists the missing engines in meta.engines_failed.

Reading the pages behind the results

A SERP gives a title and two lines of snippet. An LLM usually needs the page. OpenSERP’s /extract?url= fetches a URL and returns its main text, language, canonical URL and Open Graph tags. In auto mode it tries a plain HTTP fetch first and starts Chrome only when the text comes back shorter than a threshold, keeping whichever version holds more text. https://go.dev/doc/ came back in 0.30 s (49 KB, plain fetch, no browser needed). Extraction quality across tools is measured in Web scrapers for local LLMs.

What breaks

• Volume from one IP. Engines count queries per address; past a threshold they do not publish, the answers turn into captchas and blocks. Spreading queries over more exit IPs is the fix, and it is the part of a SERP setup that costs money.
• Ecosia returned 403 on a one-query smoke test (golang) and was not benchmarked here.
• Terms. Google’s robots.txt has Disallow: /search for all user agents, and its terms of service forbid automated access that ignores robots.txt. Running a SERP scraper is a risk you carry, at your own volume.

Self-hosted vs a hosted SERP API

An engine’s own API, such as Brave Search’s, is a third option: no scraping at all, results from that engine’s index only. How search results compare in quality is in Web search quality for local LLMs; what a search response costs in tokens is in Output formats & token cost.

ai llm engineering