Legora's P99 search latency hit 20 seconds before a redesign

Legal AI platform Legora tells the story of a search architecture that broke at scale: partitioning documents by project across 4,000 Postgres partitions looked reasonable on paper but thrashed the cache once dead projects and live projects landed together. The fix, moving to turbopuffer with one namespace per project, overturns the assumption that finer-grained sharding always helps — sometimes the unit of storage matters more than the partition count.

• Legora's workloads — Legora runs two search types: project search (tens to millions of documents per client project, e.g. an M&A deal) and legal research, a deep-research-style workload spanning laws, cases and regulations across jurisdictions.
• Gen 1: one cluster — Legora started with a single Elasticsearch cluster and one blob store for all tenants and all documents.
• Gen 2: regional split — Clients in the US, EU and Asia Pacific each demanded in-region processing, so Legora multiplied the entire stack by three or four, one Elasticsearch setup per region.
• Enterprise demands — Big banks and top law firms demanded full physical isolation and customer-managed encryption keys, letting them revoke key access to make their data unreadable.
• Gen 3: Postgres — Legora moved search into Postgres (pgvector with disk ANN and ts_vector instead of BM25) since it already ran Postgres for OLTP, partitioning document chunks into 4,000 partitions binned by project key.
• The break — Dead projects and hot projects landed in the same partitions; every query pulled a huge partition into memory and evicted others, thrashing the cache and pushing P99 search and ingestion latency from 100 milliseconds to 20 seconds.
• Move to turbopuffer — At roughly 400 million documents Legora switched to turbopuffer with one namespace per project, gaining real BM25, lower latency, single-cluster simplicity, and near-zero cost for idle projects sitting in blob storage.
• turbopuffer's design — turbopuffer writes directly to object storage with no disk replication or Paxos, targeting around three round trips per query against S3's ~200ms P99 for a 1MB blob.
• Encryption via namespace — Because the namespace is the atomic unit, each one can sit in its own bucket with its own encryption key, satisfying banks' demand for physical isolation; turbopuffer even disabled its NVMe SSD cache for some Legora workloads after testing showed memory-cache-only performance was good enough.
• Latency results — After the migration, median project-search latency improved by an order of magnitude, with P99 improving even more.
• Legal research at scale — Legal research is racing toward 10 billion vectors, fans out into many filtered queries to respect jurisdictional hierarchy and temporal validity of rulings, and now uses jurisdictions as namespaces — hot ones like EU law stay cached, cold ones like Danish law sit on blob with acceptable 500ms latency.
• Storage engineering crash course — Simon explained turbopuffer organizes vectors into a tree of clusters (not a navigable graph) to minimize costly round trips to object storage, and described full-text search as a hashmap-and-intersecting-sets process scored by BM25 that is computationally harder than vector search at web scale.
• Results claimed — Legora now runs 70-plus tenants (with figures cited up to 200) on turbopuffer without separate Elasticsearch databases per tenant, which Jacob says would otherwise be operational hell.

In their words

So we went from like search and ingestion P99 of 100 milliseconds into 20 seconds, which you can imagine is a really bad user experience.6:15

So then we went to turbopuffer in about you know when we're about I think 400 million documents something like that and what we did with turbopuffer was we did one name space per project6:20

what we did was that we thought we were going to implement encryption into the disc cache but instead we just disabled the disc cache and saw how it fared and the performance of turbopuffer even without the disc cache with just the memory cache was so good that we just kept it that way10:47

counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search.18:52

Disclosure · Both speakers represent companies with a commercial relationship: Jacob Lauritzen is an engineer at Legora, a paying customer, and Simon Eskildsen is CEO/co-founder of turbopuffer, the vendor being showcased.

One thing to add — One thing to add — the talk is light on independent verification; all latency and scale figures (100ms to 20 seconds, 400 million to 10 billion vectors, 70+ tenants) come directly from the vendor and customer with no third-party benchmark cited. The disabled-SSD-cache anecdote is presented as informal testing rather than a rigorous study.

One thing to try tonight
Check whether your own sharding scheme co-locates "hot" and "cold" data in the same partition — a quick audit of query frequency per partition key can reveal the same cache-thrashing pattern Legora hit before it causes a latency spike.