Date: September 23, 2025

Topic: Fine-Tuning

Recall

Notes

Fine-Tuning

• Pre-trained language models are trained on large, general corpus of text and are generally capable of generating fluent text
• To specialize the output, one can fine-tune the model
  • Done by continuing to train the pre-trained model with a small, specialized corpus
• This shifts the weights in the model, biasing it toward new data

When to Fine-tune

• The fine-tuning dataset is too small to train a fluent model by itself
• The fine-tuned model will start to specialize its responses without losing the ability to respond to broader contexts
• Nothing additional is needed, except the pre-trained model and new dataset

Through fine-tuning, we introduce examples that bias the model towards the fine-tuned dataset.

GPT2 Fine-Tuning Example

Without Fine-tuning

After Fine-tuning on Tolkien’s Silmarillion (Never uses the word “fox”)

• A lot of fantasy elements ended up being introduced to the text

Through fine-tuning, we are able to change the distributions of the predicted probabilities. For example, instead of the word “company”, the word “elves” is now more likely.

Encoder/Decoder Computation Graph

• When doing fine-tuning, we see that the $y$ outputs end up being words that are more for general use rather than fantasy

  • E.g., input “In that time the” most likely outputs “company” due to training on the large corpus, but the fantasy text has the output “elves” instead

    

  • The model prefers standard words like “company” and “world”, with “elves” being ranked relatively low

  • However after fine-tuning, we may get a distribution like this instead, where “elves” is ranked much higher

    

Prompting vs. Fine-Tuning

• For very large models, it is possible that the model has already seen the style we want it to emulate
• In that case, we can just prompt the model to follow a style instead of doing fine-tuning

<aside> 📌 SUMMARY: Fine-tuning is useful when we have a small dataset that we want the model to bias to, but not large enough to train a model from scratch. This helps drive the distribution of outputs towards that dataset. If we have very large models, chances are such specific datasets are already included in the model’s distribution and through careful prompting, we can drive it towards those outputs instead.

</aside>

Date: September 23, 2025

Topic: Instruction Tuning

Recall

Instruction tuning is a form of fine-tuning that aligns the model towards a question-answer format, with humans correcting the output so the model is less likely to treat the prompt as a text continuation problem.

<aside> 📌 SUMMARY: Instruction tuning helps align the model towards human conversation structure instead of being solely text generating.

</aside>

Date: September 24, 2025

Topic: Reinforcement Learning