Complete, or partial and pretending

"Extract every sentence that mentions Mr X, or refers to him in any way." A partner asked us that during a feedback session, across a few hundred documents. The system returned about ten passages, all of them correct, and none of them wrong.

It was also, we are fairly sure, incomplete. And nothing in the answer said so.

Two different operations wearing one name

Retrieval-augmented systems are built around a ranked search: score every candidate against the query, take the best k. That is exactly right for "what does this contract say about termination", where you want the strongest passages and more of them would only add noise.

It is exactly wrong for "all". "All" is not a ranking question, it is a filter-and-collect question: which units match? — return every one. Answering it with a ranked top-k produces something with the shape of an answer and no relationship to completeness, and no amount of raising k fixes it, because trusting a similarity score to have put everything near the top is still trusting a similarity score.

What everyone else does

The serious tools in this market do reach completeness, and they reach it the same way: run a language model over every document, one at a time, and collect the results. It works. It is also O(documents) model calls per question, its recall is bounded by how well each per-document extraction went, and nothing it learns is reusable by the next question.

We built the other thing. An access-scoped sweep over the index returns every matching unit deterministically, at close to zero marginal cost per query, and it is auditable in a way a fan-out of model calls is not — you can point at the query and the corpus, not at three hundred independent judgement calls.

The manifest is the feature

The retrieval change was the easy half. The half that mattered was deciding what a sweep owes you when it cannot read something.

Every run publishes a per-document manifest: scanned, awaiting index, or failed. A document still being processed is waited on and resumed — never skipped, because a sweep that quietly dropped a file would be indistinguishable from a complete one, which is precisely the failure we were trying to kill. A scanned PDF whose OCR failed is reported as failed, by name.

So the output of "read all 314 documents" is not a list of hits. It is a list of hits plus the sentence "312 of 314 read", plus the two that were not, plus why. That second part is the deliverable. It is what lets someone say to a client, with a straight face, exactly what was reviewed.

Following the reference, not just the name

"Or refers to him in any way" is the harder clause in that original request. A sentence saying "the Purchaser shall indemnify" contains none of the query's words.

Contracts, fortunately, are unusually well-behaved here: they define their own terms. So the first resolution layer is deterministic — parse the definitions, build the alias map, and "the Purchaser" reaches the same entity as the party's name with no model involved and no confidence score to argue about. Every mention records how it was resolved, so a reader can filter by provenance rather than taking the whole set on trust.

Pronouns and looser nominal references are genuinely harder and we are not going to overclaim them. What we will claim is the part that matters most in legal work: when the system says it read everything, there is a document you can open that says what "everything" meant.