This chapter was about checking whether our search results are actually useful.
We started with manual review, then added a golden dataset and metrics so we could compare results consistently. We also used debug logging to trace search problems and experimented with an LLM rating result relevance.
Manual evaluation means running test queries, looking at the results, and asking whether they are useful.
The main questions were:
A result can be technically related to a query without being a good result. Manual review helps catch problems that a numerical score might miss.
A golden dataset is a set of test queries with known relevant results.
It gives us the same examples to test against whenever we change the search system. This makes it easier to compare different approaches and see whether a change helped.
The relevant results should be chosen by someone who understands the domain. The dataset will not be perfect, but it should reflect realistic searches.
Precision@K tells us how many of the top KK retrieved results are relevant.
precision = relevant retrieved / total retrieved
If 7 of the top 10 results are relevant, the precision is 0.7.
High precision means there is less irrelevant material in the results.