Overview

This chapter combined the two search systems from earlier in the course, BM25 keyword search and chunked semantic search, into one hybrid system. Neither method works well for every kind of query, so the idea is to run both and merge the results. I built two different ways to do that merge: weighted score combination and Reciprocal Rank Fusion (RRF).

Key Concepts

Keyword vs. Semantic Tradeoffs

Quick breakdown of when each one wins:

• Keyword search: good for exact titles, years, names, anything where the exact word matters.
• Semantic search: good for concepts and themes, like "family friendly" or "survival movies," where the exact words might not appear in the text.

Neither one covers the other's weak spots. That's the whole reason hybrid search exists.

Query Categories

Queries roughly fall into three buckets:

• Known-item: user wants a specific title. Keyword usually wins.
• Topical: user describes a theme or concept. Semantic usually wins.
• Exploratory: user is just browsing. Could go either way.

Score Normalization (Min-Max)

BM25 scores and cosine similarity scores are not on the same scale. BM25 can go well above 1, cosine similarity is always between 0 and 1. You can't just add them together or compare them directly, so you need to normalize first.

Min-max normalization squashes a list of scores into a 0 to 1 range:

normalized = (score - min_score) / (max_score - min_score)

The lowest score in the list becomes 0, the highest becomes 1, everything else lands in between. It works fine but it's sensitive to outliers since one huge score can compress everything else toward 0.

Weighted Combination (alpha)

Once both sets of scores are normalized, you can blend them with a weight called alpha: