Overview

In Chapter 4, we turned entire documents into single embedding vectors. That worked fine for tiny samples, but longer texts run into limits. If a document covers multiple topics or runs too long for an embedding model, turning the whole thing into one vector muddies the meaning.

This chapter is about chunking: breaking longer documents into smaller pieces before we embed them. In our RAG pipeline, this lets us search for the specific part of a text that answers a query instead of having to feed huge, noisy documents into an LLM later.

Key Concepts

• Chunking: Splitting a document into smaller text blocks before generating embeddings. Smaller pieces make it much easier to match specific facts.
• Chunk Size Trade-off: If chunks are too small, you lose the surrounding context needed to understand the idea. If chunks are too large, the specific details blur together, and you risk hitting model input limits.
• Chunk Overlap: Repeating a bit of text from the end of one chunk at the start of the next one. This prevents an important sentence or idea from getting cut right in half across a boundary.
• Fixed-size vs. Semantic Chunking: Fixed-size chunking splits strictly by a count (like characters or words), which can cut sentences off mid-thought. Semantic chunking tries to split on natural boundaries, such as sentences or paragraphs, to keep coherent thoughts together.
• Mapping Chunks to Documents: When we search, we compare our query vector against individual chunk vectors. We still need to know where each chunk came from, so every chunk stores a reference back to its parent document.
• Deduplication (Best Chunk per Document): If a document has several chunks that match a query, we do not want that single document filling up all top search slots. We group results by document and keep each document's highest-scoring chunk to rank the documents fairly.
• ColBERT & Late Chunking (Advanced Concepts):
  • ColBERT: Keeps token-level embeddings and compares query tokens to document tokens directly (late interaction) instead of squishing everything into one vector.
  • Late Chunking: Passes the whole document through a long-context embedding model first so tokens see the full context, and only pools them into chunk vectors afterward.
  • Note: These are advanced methods covered conceptually. You only need them if basic chunking is not accurate enough and you have the extra compute to spare.

What We Implemented

• Added chunking helpers to split document text into smaller chunks using target sizes and overlap.
• Swapped our indexing step from embedding whole documents to embedding individual chunks.
• Updated semantic search to:
  • Embed the query.
  • Compare the query embedding against all chunk embeddings.
  • Map each chunk match back to its original document ID.
  • Deduplicate results by keeping only the highest-scoring chunk for each document, then sorting those documents by score.

Why It Matters

• Better Match Quality: A user query usually targets a specific answer, not a 10-page document. Chunk embeddings make targeted semantic matches much cleaner.