This chapter introduced two things: a faster way to find documents, and a way to rank them.
Before this, search was brute force. Loop over every query token, loop over every movie, check for matches. That works but it's slow, and it doesn't tell you which results are actually good.
So we built an inverted index to make lookups fast, then added TF, IDF, and TF-IDF to score how relevant a document is for a given term.
Inverted index
A normal index maps location to value. An inverted index goes the other way, mapping value to location.
In our case each token maps to the set of document IDs that contain it:
matrix → [1, 5, 10]
Looking up a token is roughly O(1). Building the index is the slow part since we have to read and tokenize every document once, but we only do that once.
Docmap
Separate dictionary mapping document IDs back to the full movie objects.
The index tells me which documents matched. The docmap tells me what those documents actually are. Keeping only IDs in the index means we're doing set operations on small integers instead of big chunks of text.
Pickle for persistence
The index, docmap, and term frequencies get saved to cache/*.pkl. The build command writes them, load reads them back. As long as movies.json doesn't change, there's no reason to rebuild.
Boolean search
Since tokens map to sets of doc IDs, boolean operators are just set operations: