RAG architecture: chunking, embedding, retrieval

The pipeline that puts the right context in front of the model instead of all of it.

6 min read · Lesson 7 of 9 in this domain

Retrieval-augmented generation exists because the alternative — putting an entire corpus in the context window — is impossible for large corpora and wasteful for small ones. The pipeline has three stages worth separating. Chunking splits documents into retrievable units, and the size is a genuine trade-off: chunks too small lose the surrounding meaning that makes a passage interpretable, chunks too large dilute the signal so retrieval matches on the wrong thing. Embedding turns each chunk into a vector so semantic similarity becomes a distance computation. Retrieval embeds the query the same way and returns nearest chunks. The quality ceiling is set at chunking: nothing downstream recovers meaning that the split destroyed.

• Chunk size is a trade-off — too small loses context, too large dilutes the signal and retrieves imprecisely.
• Overlap between adjacent chunks preserves meaning that would otherwise be cut across a boundary.
• Prefer splitting on natural structure — sections, paragraphs — over fixed character counts that cut mid-sentence.
• Embeddings turn semantic similarity into vector distance; the query is embedded with the same model as the chunks.
• Retrieval quality is capped by chunking. No reranker recovers context the split removed.
• Retrieved text is untrusted input: tag it and instruct the model to treat it as data, not as instructions.
• More retrieved chunks is not better — you are spending context, and the lost-in-the-middle effect applies to what you inject.

Source: Claude Docs — Context windows · Independent study aid, not affiliated with or endorsed by Anthropic.