A document platform for humans and agents: the architecture
A corpus of unstructured documents that people and agents query through one index. Six stages, six stores.
Six stages, six stores. The only loop: synthesis made in Products re-enters Ingest as a new document.
Stages
Stage
Does
Store
Sources
pulls feeds, files, APIs and internal notes on a schedule or by push; each source has an owner
raw store, byte-exact, keyed by content hash
Ingest
parses to text, strips boilerplate, dedupes by hash, keeps every version under one doc_key
document store
Enrich
tags each document against the ontology: entities, countries, concepts, markets, and the period it discusses
tag store
Index
splits into chunks, embeds them, writes text, vector and filter fields
search index
Access
one query API with two profiles (human, agent), entitlements and per-agent budgets
budget store (sessions, quotas)
Products
saved searches, collections, feeds, cards; writes synthesis back
user state
Every document gets a schema, an owning source, ontology tags and versions instead of overwrites.
Schema
The chunk is what the index returns. Its lower part is copied from tag, entitlement and document rows when it is written, so no filter needs a join at query time.
tag.kind is a closed list; tag.value is an ontology id, never free text. Tags under a confidence threshold are not copied to chunks.
period_from / period_to describe what the document is about, separate from when it was published.
Each chunk copies tags from its document’s tag rows, acl (group ids) from entitlement rows, and published_at from the document. A tag or entitlement change rewrites those fields on the document’s chunks; until the rewrite lands, the API re-checks entitlement rows for every returned chunk.
valid_from is this version’s ingested_at; valid_to is the next version’s, or open.
card.evidence is a list of chunk ids, not a foreign key: a card keeps pointing at the chunks it was built from.
Chunking: about 1k tokens, split on headings and paragraphs, 10–15% overlap.
Search
Post-filter asks the index for k and throws away what fails the filter: when 7 of 10 fail, the caller gets 3. Native filtering checks the filter on each hop of the vector graph walk; when a filter leaves very few candidates the walk loses its way, so the engine switches to an exact scan of that subset.
Almost every query carries a date, source, entity or ACL condition, so filters run inside the vector search.
Each query runs a keyword search and a vector search over the same filtered chunks. The two rankings are merged with reciprocal rank fusion; the human profile reranks the top 50 with a cross-encoder.
Point-in-time
Each ingest of the same doc_key is a new version; valid_from is its ingested_at, valid_to the next version's. A query sees the one version valid at its as_of, so dated questions and backtests are reproducible.
Every query takes an as_of. A chunk is visible when valid_from ≤ as_of < valid_to, so a dated question or a backtest sees each document as it was then and nothing ingested later. ingested_at is set by ingest and never backdated; that keeps as_of results reproducible. The tag period (period_from / period_to) is a separate filter on what the document is about, not on when it was ingested.
Two query profiles, one index
Human profile
Agent profile
Reads
~300 words/min
tens of thousands of words/min
Optimised for
precision
recall
Results per query
10, reranked
50–200 chunks
Returned text
title + short snippet
full chunk text + ids for citation
Plan for
~100 queries/hour in total
~10 queries/s per agent, 1,000+ for a swarm
Limits
none
rate limit and budget per agent
Same index, filters and ACL; only k, reranking and payload differ. The agent profile sets the scale:
Shard by source, so date, source and tenant filters skip whole shards; hash sharding would send every query to every shard.
Add read replicas for query volume.
Cache agent results by (profile, k, query, filters, as_of rounded to the minute).
Quality
Labels are document ids, so they survive re-chunking. The score runs in CI next to the unit tests.
Products
Saved searches run in reverse: each new chunk is matched against all active saved searches as it is indexed, so cards appear without re-running searches. A synthesis is a document like any other, so the watchers see it too.
Collections are fixed document sets a user hands to an agent as its corpus. A synthesis note gets the intersection of its evidence chunks’ ACLs, so a summary never reaches someone who could not read its sources.
Access
Every client, agents included, goes through the same API. The caller's groups become an acl filter inside the index query; agent budgets are checked here too.
Clients never query the index directly; budgets and the acl filter live in the API only.
Build order
Raw store, ingest, document store with doc_key, versions and hashes.
Ontology and tagger; tag store.
Chunking, embeddings, index with native filtering and the copied filter fields.
Query API with both profiles, as_of and entitlements.