People treat retrieval like a plumbing choice: pick an embedding, pick a vector store, tune k. That is the last ten percent. The first ninety is deciding what a record is.
If you chunk a hundred-page manual by token count, you will retrieve a paragraph that mentions the right noun and the wrong rule. The model will not know. It will sound sure. That is not a prompt problem. That is a grain problem.
I start from the questions the system will actually get asked, then I work backward to the unit of evidence that can answer them. Sometimes that unit is a section with its heading path intact. Sometimes it is a row — a coverage, a location, a storm, a timestamp — with the surrounding prose attached as context, not the other way around. Hybrid search is useful. It does not excuse a chunk that has no identity.
Metadata is the other half. Source, as-of date, jurisdiction, whether the text is binding or commentary. If you cannot filter before you embed-rank, you will retrieve last year’s exception and this year’s rule in the same context window and call it grounded.
I also keep a holdout of questions whose answers I can point to by hand. When retrieval quality drops, I want the miss: wrong document, right document / wrong span, or the thing was never in the corpus. Those three failures want three different fixes. Collapsing them into “RAG is unreliable” is how teams add another reranker and change nothing.
The model is the last reader. If the packet you hand it is a junk drawer, it will rummage. Retrieval is data modeling with a latency budget. Treat it that way and the embeddings finally have something to do.