This chapter added image input to search. Before this, everything was text in, text out. Now we can search using a picture instead of typed words.
There were two ways to do this. First, use an LLM to turn an image into a better text query. Second, embed the image directly and compare it to text embeddings using cosine similarity, same as regular semantic search. This chapter walks through both, but the real implementation uses the second approach.
Multimodal query rewriting
The simple version: send an image and a text query to a multimodal LLM, and ask it to describe the image and rewrite the query into something more searchable. The output is just a better text query. It doesn't search anything by itself, you still need to take that rewritten query and run it through your existing search pipeline. It's a quick way to add image support without changing your embeddings or search logic at all.
The downside is it's lossy. You're compressing an image into a sentence or two, so you lose whatever detail didn't make it into the description.
Shared embedding space
The better version skips the "describe the image in words" step entirely. Instead, both images and text get turned into embeddings that live in the same vector space. That means you can compare an image embedding directly to a text embedding with cosine similarity, the same math from earlier chapters, just applied across two different types of input.
Contrastive learning (just enough to understand why this works)
The reason image and text embeddings can even be compared is how the model was trained. It was shown matching pairs (an image and its caption) and trained to push those embeddings close together, while pushing unrelated pairs apart. That's basically it. It's why "a photo of a bear" ends up near an actual bear photo in vector space, instead of the two living in unrelated spaces.
CLIP
CLIP is the model family we used to get this shared space. It has separate encoders for images and text, but they were trained together so their outputs line up. We used a small, local version of it (clip-ViT-B-32) through sentence-transformers. It's good enough to experiment with and learn the concepts, but it's not a huge or particularly powerful model.
Combining image and text (hybrid queries)
Since images and text share a vector space, you can also blend them, like take 70% of an image embedding and 30% of a text embedding to build one combined query vector. This was mostly conceptual in this chapter, just a "here's what's possible" example, not something we built out fully.
I built describe_image_cli.py first, which does the query rewriting approach. It sends an image plus a text query to a multimodal LLM through OpenRouter, and gets back a rewritten text query along with token usage.
Then I built the real image search feature. I created a MultimodalSearch class that loads a CLIP model and can embed images. I first checked that it actually produced embeddings and printed the dimensions to sanity check it.
After that, I extended the class to also embed the text side of the movie dataset, titles and descriptions combined, so there'd be something to compare images against. Then I added an image_search command: you give it an image path, it embeds the image, compares it to every movie's text embedding with cosine similarity, and returns the top 5 closest matches with their similarity scores.