This chapter is where the Webflyx search project gets started. The setup is a CLI that searches a local movie dataset in data/movies.json.
We began with the dumbest possible version of search (does the query show up literally in the title?) and then kept improving it with a text processing pipeline.
In RAG terms (Retrieve, Augment, Generate), this is the very front of the "R". No scoring, no embeddings, no LLMs yet. Just getting raw text into a shape where comparisons actually work. Everything later in the course assumes this step already happened.
Search as a feature
Users don't type exact matches. Somebody searching "The Alien" probably still wants "Alien" and "Aliens" and "the alien". So the system has to be forgiving about case, punctuation, and word forms.
Keyword search
Check if the user's terms appear in the data, return what matches. It's fast and simple, and it's genuinely good when the vocabulary is precise. Searching "COVID-19" in medical documents works great because there's one right word.
Where it falls down is generalization. "Alien movie" finds "Alien" and nothing else, even though "E.T." is arguably what the person wanted.
The text processing pipeline
Five steps, applied in order:
Bear and bear become the same keyword.sci-fi turns into scifi, which is a known downside with hyphenated words.bear token. Worth remembering this is not the same thing as LLM tokens.a, the, is, of. Skip this and a search for "the bear" returns "The Terminator" right alongside "Jungle Bear".running and runs both become run. Without it, searching "running" misses "A River Runs Through It".Process both sides the same way
The query, the movie titles, and the stop word list all need identical treatment. The stop word file is the easy one to forget. aren't has to become arent or it will never match a processed token. If the two sides are processed differently, matches just quietly disappear.