Web search quality for local LLMs: did the search engine find the right answer?
Search quality = did the query return the page that answers it. Best published: Brave 94.1%, Tavily 93.3% (vendor-run SimpleQA). SearXNG on robertz.co: q2/q3 each returned the target page; bare robertz.co 2 of 10 on-target.
Series — web search for local LLMs: Overview · Native Claude vs self-hosted · Search quality · Output formats & token cost · Next stage: Web scrapers for local LLMs
Section A is published third-party benchmarks. Section B is our own spot check on robertz.co. How cleanly a page’s text is extracted after it is found is the next stage — Web scrapers for local LLMs.
A · Published search accuracy
Almost every number here is vendor-run SimpleQA on the vendor’s own configuration — directional, not a fair head-to-head. SearXNG has no benchmark at all: it is the pipe, so its ceiling is whatever Brave + Google CSE return upstream.
| Engine / product | Answer accuracy | Run by | Also reported | Source |
|---|---|---|---|---|
| Brave Search API (“Answers”) | 94.1% F1 | vendor | 76.05% in Tavily’s independent run | brave.com/blog/ai-grounding |
| Tavily | 93.3% SimpleQA | vendor | Document Relevance 83.0% | tavily-ai/tavily-search-evals |
| Perplexity Sonar Pro | 0.858 F (SimpleQA) | vendor | Sonar 0.773 | perplexity.ai/hub/blog |
| Google SERP via Serper | 82.15% SimpleQA | — | — | tavily-search-evals |
| Exa | 71–91% SimpleQA (by setup) | vendor | FRAMES 54.4% | exa.ai |
| SearXNG | no benchmark — = upstream engines | — | it’s the pipe (Brave + Google CSE), not a graded product | — |
| Native Claude web search | no plain SimpleQA published | vendor | dynamic filtering: BrowseComp 45.3→61.6% (Opus), DeepsearchQA F1 69.8→77.3% | claude.com/blog/improved-web-search-with-dynamic-filtering |
B · SearXNG vs native Claude on robertz.co (our run)
Run locally on my fleet — one site, not a benchmark. Native = Claude Code’s WebSearch tool (Anthropic server-side search), result list as the tool returns it.
| Search query | SearXNG on-target | Native on-target | Target found (SearXNG / native) | SearXNG hit | Native results |
|---|---|---|---|---|---|
q1 robertz.co | 2 of 10 | 0 of 10 | homepage #3 / ✗ | robertz.co/ and /simulators/ — other 8 look-alike domains | auditregister.org.uk ×5, finmag.cz ×2, ltds.uk, officer.com, thewaternetwork.com |
q2 robertz.co authentication methods battle | 3 of 10 | 0 of 9 | ✓ #2 / ✗ | the auth-methods article | generic auth pages: academia.edu, thalesgroup.com, scribd.com ×3, oreilly.com, … |
q3 robertz.co broker_parrot self-hosted job fleet | 2 of 10 | 0 of 10 | ✓ #1 / ✗ | the broker_parrot post | look-alike job brokers: pypi.org/jobd, producthunt.com, glama.ai, … |
| latency | 0.89 s warm · 4.02 s cold | 5.62 s (incl. Claude Code tool overhead) | — | — | — |
Caveats
- Section A is directional. Cross-vendor SimpleQA uses different judges, prompts and configs — a vendor grading its own “best” product is not a head-to-head. Setup alone moves Exa 20 points and Brave from 94 to 76.
- SearXNG has no benchmark. It is a metasearch pipe; its ceiling is the upstream engines it queries.
- Section B is one site. Three queries on robertz.co — a spot check, not a ranking.