This is going to be your holy grail to understand llms 😊

We are going to understand the llms while building one

pipelines /stages for building llms

1.pretraining(

1. download and process the internet data
2. tokenization
3. neural network training ( in this step we want to modeled the statastical relationship of how this tokens in the dataset follow each other in the sequence)

)

2.mid training

3.fine-tuning

common crawl, seed webpage, fineweb blog post

firstly data is collected from web crawl

crucx of how text data is converted to what llms understand:

the data is converted to utf-8 which is a unicode standard (it is a standard where the characters are mapped to the ids(numbers) for computers to understand )

after converting to the utf-8 it is then using Byte Pair Encoding Algorithm that decreases the sequence length

why decreasing the seq length? because it is a important or valuable resource that can bloat models if too large

decreasing the seq length comes with increasing in vocab size (it is the raw tokens model know’s )

After all of this the text is converted to the token id’s

After that we start with the neural network training or model training

we do it in windows that is we select chunks(any size from 0 to the max size we decide on this become the maximum context length of the model and can randomly choose) from all of the data… the greater the window size the more the compute..

now we feed the sequence of tokens we have picked to the neural network and what it does ? it ouputs all the tokens we have in our vocabulary with probability of which one could be the next

and lets say out input is “the cat sat on the ” and the model output the probability “mat” more than anyone than it is correct (i.e it is statistically match the data) if it doesn’t we’ll use a algo to feed it the correct output so it get’s align (match the pattern ) with the dataset statistically..

NOTE: input and output is not text it is ‘id’s’ we have converted our data into