Why AI document search breaks down at 10,000 files.

AI search can look impressive with 50 documents and fail with 10,000. Reliable results require careful engineering.

We get a version of this email at least once a month. The demo was extraordinary. The founder stood up at the all-hands, asked the assistant a question, and it pulled the right answer out of fifty internal PDFs. It looked like magic, and the board signed off. Six weeks later they had loaded ten thousand real documents into the same pipeline, and the assistant was confidently answering a renewals question with policy text from a dead 2019 deck.

What changed? Nothing changed in the architecture. That's the whole problem. RAG — retrieval-augmented generation — means finding the relevant text first, then letting the model answer from it. The naive version has roughly the same shape at fifty documents and at fifty thousand: chop documents into chunks, turn each chunk into a list of numbers, fetch the closest few, paste them into the prompt. The failure modes, though, are not linear. They're cliff-shaped.

This is a field guide to those cliffs. It's the talk we give clients at the start of a RAG engagement, written down so we can stop giving it on calls.

The demo lied to you

On a set of fifty hand-picked PDFs, almost any retrieval method works. The question and the answer usually sit close together in the text, and they use the same words. There is exactly one document that could plausibly answer any given question. Fetching the four closest chunks will hit the right one most of the time, and the model's reasoning covers the rest.

At ten thousand documents, every one of those assumptions is false. Now seventeen documents could plausibly answer the question: three out of date, two contradicting each other, one written for a different country. Shared vocabulary collapses, because users ask in their words and your documents are written in your team's words. Meanwhile the similarity maths keeps working exactly as designed, but the space around every query fills with plausible-but-wrong neighbours — chunks that look like the question without being evidence for the answer. Similarity was never the same thing as relevance; scale is just when you find out.

In our experience, the gap between "the demo passes" and "the right document lands in the top ten results more than 70% of the time" is roughly ten times the engineering effort. Every published benchmark we have seen points the same way. People wildly underestimate this, because their first build was so easy.

Chunking is not splitting

The first thing that breaks is chunking — how you cut documents into the pieces you index. The stock LangChain recipe (RecursiveCharacterTextSplitter, 1,000 characters, 200 of overlap) is fine for blog posts. It is catastrophic for invoices, contracts, manuals, and anything with structure. It chops tables in half. It detaches a section heading from its body. It produces chunks that are grammatically complete and meaningless: "…and shall be governed by the laws of the State of", with the rest of the clause living in the next chunk.

What works at scale is some form of structure-aware chunking: read the document with a tool that understands its layout, cut on natural boundaries (sections, list items, table rows), and carry the surrounding context with each piece. The technique we lean on most is contextual chunking — before indexing a chunk, prepend a one-line, model-written note saying which document and section it came from. Anthropic's contextual-retrieval study put numbers on it. That prefix alone cut top-20 retrieval failures by 35%. Pairing it with a keyword pass took the reduction to 49%, and adding a reranking stage on top reached about 67%. Those are three different setups, and it's worth knowing which one you are building. We see improvements in the same territory on real client document sets; it is the single highest-leverage change we make.

For tables and code, you don't chunk at all in the conventional sense — you serialize them as structured records and index them separately, with their own retrieval pathway. A single misshapen chunk of a fifty-row pricing table will quietly poison answers for months.

Embeddings have a domain ceiling

An embedding model turns a piece of text into a list of numbers so that similar meanings land near each other. The off-the-shelf ones — OpenAI's text-embedding-3, Cohere's embed-v4, Voyage's voyage-4 family — are general-purpose by design, tuned to handle broad, common language well. What they are not tuned for is your language. They are, in the polite phrasing, less calibrated on legal opinions, claims-adjuster notes, surgical reports, chip datasheets, and the in-house vocabulary of any company older than ten years.

You can see this in two symptoms. Rare terms miss: a user types a part number or an internal acronym, the model has no feel for it, and you get back documents that are vaguely on topic and simply not the one. Mixed-language questions miss harder. A support query written in Hinglish, common in our local market, often pulls back documentation in the wrong language entirely.

The fix is not to chase a slightly better embedding model. The fix is to stop relying on embeddings alone.

Top-k is a polite lie

"Fetch the top ten results" sounds like enough until you measure it. Across our own client engagements, the right document lands somewhere in that top ten between 45% and 65% of the time. That figure comes from labelled test sets of 100–200 question-and-correct-document pairs per project, so treat it as Script Lanes internal data rather than a published benchmark. It means that more than a third of the time, none of the ten chunks the model is reasoning over actually contain the answer. The model then builds something plausible out of the wrong context and ships it as truth. That is not a hallucination. That is a retrieval failure wearing a hallucination's clothes.

The pattern that consistently moves the number is over-fetch, then re-rank. Pull 50 to 100 candidates from the cheap first pass, then run a second, slower model over them to pick the 5 to 10 the assistant actually sees. That second model — a cross-encoder — reads the question and the document together instead of scoring each separately, which makes it slow per pair and far more accurate. It isn't a law of nature, but for serious production search it has become hard to justify leaving the re-rank stage out.

Hybrid retrieval is the actual baseline

The single most reliable architectural choice we make on production RAG is hybrid retrieval. Run one pass on meaning, using the embeddings. Run a second pass on keywords, using BM25, the scoring method behind classic search engines. Merge the two lists — usually with Reciprocal Rank Fusion, which needs no tuning — then re-rank the merged set with a cross-encoder.

BM25 catches what embeddings miss: exact part numbers, error codes, proper nouns, the legalese a customer literally pasted from your own documentation. Dense retrieval catches what BM25 misses: paraphrases, synonyms, "how do I cancel" matching a doc titled Subscription Termination Policy. Either alone is brittle. The two together are robust in a way neither's authors will tell you about, because each of them is selling one half of the answer.

Concrete recipe we ship by default: keyword + meaning → fuse → re-rank → top 5. The components matter less than the shape. We use Qdrant or pgvector for the meaning pass, Elasticsearch or a Postgres full-text index for the keyword pass, and Cohere Rerank or bge-reranker-large for the re-ranker. Swap any of those without changing the shape and your numbers move within a small band.

You cannot improve search without measuring its accuracy

Here is the most common failure mode we walk into: a team has been iterating on prompts and chunking for three months, has no idea whether their system has gotten better or worse, and is operating entirely on vibes from the latest demo. There is no eval set. There is no recall number. The product manager's gut feel is the regression suite.

Before any of the work above is worth doing, you need a retrieval test set: a hundred to two hundred pairs of question and correct document, taken from real user behaviour or, from a cold start, written by people who know the material. Then measure three things. Recall@k: was the right document in the top k results? MRR: how high up the list did it appear? NDCG: the same idea when some documents are more relevant than others rather than simply right or wrong. Those numbers tell you whether retrieval is improving. The model's final answer does not.

On top of retrieval evals, you want citation evals: did the answer the LLM produced actually use the documents that were retrieved, or did the model wander? The cheap version is to require the model to emit citation IDs and check that the cited chunks support the claim. The careful version uses an LLM-as-judge with a clear rubric. Both beat "the answer looked good in Slack."

Index hygiene at scale

At 10,000 documents you almost certainly have duplicates and near-duplicates — old versions of the same policy, three slightly different copies of the same FAQ, a marketing page that has been rewritten four times. Each of those duplicates draws probability mass away from the canonical source and toward whichever copy happens to have the highest cosine score for a given query. The user's experience is that the assistant gives subtly inconsistent answers to the same question across days.

The cure is unglamorous and effective. Remove duplicates as you index, using a fingerprinting method such as MinHash or SimHash. Version your documents explicitly, and mark old versions as retired rather than deleting them. Give recent documents a scoring boost where recency matters. For knowledge bases that turn over fast, set a re-indexing schedule and a target for how stale content is allowed to get. The team that owns the index has to own the editorial state of the library, not just its bytes.

Most queries don't deserve a single retrieval

Real user queries are messier than the eval set you wrote. They contain typos, they pile multiple questions into one sentence, they reference internal jargon the user assumed you'd understand. Single-shot retrieval on the raw query underperforms badly here.

Three techniques are worth knowing. Query rewriting: a model tidies the question and expands acronyms before it reaches the search step. Query splitting: the model turns one messy question into two or three simpler ones and you combine the results. HyDE: the model invents a plausible answer, then searches using that instead of the question. The invented answer is wrong, but it reads like the documents you are looking for, so it lands nearer to them. None of these is exotic any more. All three can lift results on difficult questions, though the gains depend on your material, so measure them on your own test set. Each costs a few hundred milliseconds and a fraction of a cent.

How we run it at Script Lanes

Our default starting point on a new engagement now looks like this. We build the test set before the index — usually 150 question-and-correct-document pairs, half sampled from real user queries where they exist, half written by a subject-matter reviewer. We chunk with structure-aware splitters and contextual prefixes. We index on meaning and keywords from day one, fuse the two lists, and re-rank with Cohere Rerank or bge-reranker. We rewrite every query, and split the difficult ones. Every retrieval is logged somewhere we can inspect it (we like Langfuse), and we review a sample weekly.

The boring parts are what keep the demo working in month nine: the de-duplication job, the freshness monitor, and the automated check that fails the build when retrieval quality drops below an agreed floor. Almost none of our enterprise clients had any of that on the system we replaced. Most of our work in the first thirty days is putting it in.

The demo is not a finish line

The seductive thing about RAG in 2026 is that the first version is almost free. Twenty lines of LangChain, an OpenAI embedding key, a cheap vector DB, and a confident demo is a Tuesday afternoon. That's the trap. The system you build on a Tuesday afternoon does not survive contact with ten thousand real documents, ten thousand real users, and the editorial entropy of a real organization.

Build for the cliff, not the demo. Build the test set first. Treat retrieval as a two-pass, re-ranked problem from day one. Treat your index as an editorial product, not a data dump. And measure recall loudly, on every change, or you will be the team that can't tell the assistant got worse until a customer does.

The good news: this is now a tractable, well-understood problem. The bad news: nobody who is selling you a "RAG-in-a-box" wants you to know that.