A 30-year-old scoring function beats retrieval baselines because the user changed

BM25 was supposed to be the boring baseline every fancy neural retriever had to beat. Jo Kristian Bergum argues that in agentic search it's winning again — not because the algorithm changed, but because the thing typing the queries did.

• Definition — Bergum defines agentic search as search inside an agent loop, requiring a capable model, a harness (tool calling or code mode), and a retrieval engine that can work at billion-document scale.
• BM25 origin — BM25 stands for "best match 25" — researchers ran a series of scoring experiments and experiment number 25 performed best, which is how the name stuck.
• The real change — BM25 itself hasn't changed; what changed is the user, since LLMs carry general knowledge of entities, companies, dates, and can fire off far more queries than a human.
• BrowseComp Plus — The benchmark has 830 riddle-like questions over roughly 100,000-105,000 web documents, each with a golden reference answer to check end-to-end accuracy.
• Context as floppy disk — Bergum compares context windows to 1980s floppy discs: about 1.4 megabytes then, roughly 350,000 tokens now before quality degrades.
• Reasoning isn't the bottleneck — Stuffing evidence documents directly into context yields high accuracy even for GPT-4, but accuracy falls once the model must use a search tool, because it now depends on query formulation and retriever quality.
• AOL query logs — AOL's leaked search logs showed humans searching in short two-or-three-word queries, and Bergum says human logs today still look about the same, while GPT-5 writes much longer, operator-laden queries (site:, phrases).
• Which BM25 — BrowseComp Plus's original BM25 baseline used inadequate hyperparameters for long documents, and more recent research shows tuning the two BM25 hyperparameters dramatically changes benchmark accuracy.
• Why BM25 wins now — Exact matching handles names, zip codes, and SKUs that embedding models struggle to encode into fixed vocabularies; BM25 is cheaper than running 8-billion-parameter embedding inference; and its matches are explainable to the model.
• Waterloo paper — Bergum cites a Jimmy Lin group paper, "Scaling Direct Corpus Interaction via Dynamic Workspace Expansion," describing a file-system-style workspace where agents get progressive disclosure of retrieved documents via titles and snippets before deciding to read more.
• Evaluation shift — Classical IR evaluation (one query, one ranked list, compute NDCG) is largely obsolete for agents that issue many reformulated queries; task success on question answering matters more.
• Hornet benchmark — Hornet's own engine, tested against anonymized competitors on 100 million web documents on a single node, shows lower latency at the same throughput, which Bergum frames as cost savings for companies building web search infrastructure.
• Four takeaways — Bergum closes with four claims: there's a more powerful user now, BM25 implementations vary so "which BM25" matters, BM25 is effective because it's explainable to the model, and combining it with grep-like literal matching makes a strong agentic retrieval paradigm.

In their words

BM25 used to be a kind of a baseline function. Any information retrieval research would include a BM25 baseline and then you would put something fancy advanced neural fancy stuff and then you would compare it with BM25.3:54

it turns out that that baseline is terrible. So when you look at fancier techniques, embedding models, what have you, um it's stands out as a much better retrieval paradigm than BM25 if you look at the original paper. But more recent research shows that the parameters that were used in the browse comp plus research paper was not really adequate to handle these kind of long documents.9:52

if you artificially just stuff the evidence documents that is needed to answer this question into the context window of the model the accuracy is really high right so reasoning is not the bottleneck6:53

but if you expose the model with a harness with a retrieval tool. That accuracy falls because it now depends on the harness. It depends on the model's ability to formulate queries and the retrieval quality of the retriever.7:15

Disclosure · Bergum is CEO of Hornet.dev, a retrieval infrastructure company for agents, and presents Hornet's own BM25 engine and benchmark results in the talk.

One thing to add — One thing to add — Bergum's floppy-disk framing and the BrowseComp Plus context-stuffing result make a clean, almost too-clean case that retrieval infrastructure remains necessary even as models improve, which happens to be exactly what Hornet sells. The claim that human query behavior hasn't changed since AOL logs is asserted rather than sourced to a specific recent study, so it's worth treating as anecdotal.</note> </invoke>

One thing to try tonight
Pull up your own agent's search tool logs (or run a quick test query through it) and compare the query length and use of operators like site: or exact phrases against a typical human search — Bergum's talk suggests tuning your BM25 hyperparameters for long, specific agent-generated queries rather than short human ones.