This chapter took the TF-IDF work from Chapter 2 and turned it into BM25, which is just a better version of the same idea. Nothing about the approach changed, but three specific weak spots in TF-IDF got fixed: the IDF formula was unstable, term frequency could grow without limit, and document length wasn't accounted for. By the end of the chapter these fixes were combined into one scoring method and hooked up to an actual search command.
BM25 IDF (a more stable version)
The old IDF formula, log(N / df), breaks in a couple of ways. If a term shows up in zero documents you get a division by zero. If a term shows up in every document, the score can drop to zero or go negative. BM25 fixes this with:
IDF = log((N - df + 0.5) / (df + 0.5) + 1)
The idea is simple: compare how many documents don't have the term to how many do. The 0.5 added to both sides is just smoothing so we never divide by zero. The + 1 at the end makes sure the result stays positive no matter what.
Term frequency saturation
With plain TF-IDF, a word that shows up 100 times counts 10x more than a word that shows up 10 times. That's a problem because a document that just repeats a word a lot isn't necessarily more relevant. BM25 caps this out with:
tf_component = (tf * (k1 + 1)) / (tf + k1)
k1 just controls how fast this cap kicks in. We used 1.5. In practice this means the first few times a word appears matter a lot, and after that each additional occurrence adds less and less to the score.
Document length normalization
Longer documents rack up more word matches just by being longer, not because they're more relevant. BM25 adjusts for this by comparing a document's length to the average length across the whole collection:
length_norm = 1 - b + b * (doc_length / avg_doc_length)
tf_component = (tf * (k1 + 1)) / (tf + k1 * length_norm)
b controls how much this matters. We used 0.75. If b were 0, length wouldn't affect the score at all. If it were 1, length would fully affect it. Longer than average documents get penalized a bit, shorter ones get a small boost.
Putting it together
Once you have both pieces, the score for one term in one document is just:
BM25 = bm25_tf * bm25_idf
For a full query, you add up this score across every query term, for every document.