basics() # Assignment 1: tokenization, model architecture, training
What are the atoms that the model operates on?
Formally: a tokenizer converts between raw inputs (bytes) and sequences of integers (tokens)
Popular tokenizer:
Byte-Pair Encoding(BPE)
Intuition: break input into frequently-occuring chunks
Efficiency lens
The dream: tokenizer-free model architectures, which operate directly on bytes
These are promising, but have not yet been scaled up to the frontier.