Date: October 10, 2025

Topic: Language Models

Recall

Language models depend on words coming before to predict the next words (generative models). They can be used in predictive typing, automatic speech recognition and basic grammar correction.

Notes

Language Models

• Allow us to estimate probabilities of sequences and more importantly, compare the probabilities

Probabilities

• The probability of a sequence can be modeled by its words where subsequent words are due to prior word probabilities
• Language models are just generative models for languages

Applications for Language Models

Predictive Typing

• Search fields
• Keyboards
• Assisted typing (e.g., sentence completion)

Automatic Speech Recognition

• How likely will the user say “My hair is wet” vs. “My hairy sweat”

Basic Grammar Correction

• $P(\text{They're happy together})> P(\text{Their happy together})$

We can introduce a context $c$ into the language model. This lets us modify the probability of generating words based on the given context.

Conditional Language Modeling

• Similar to a language model, but conditioned on extra context $c$

Applications of Conditional Language Modeling

• Topic-aware language models:
  • $c$: Topic
  • $s$: Text
  • If the context is on car repairs, the probability of a text “I eat an apple” will be much lower than without the context
• Text summarization:
  • $c$: A long document
  • $s$: Summary
• Machine translation:
  • $c$: French text
  • $s$: English text

Outside of Language Modeling

• Image captioning:
  • $c$: An image
  • $s$: Image caption
• Optical character recognition:
  • $c$: Image of a line
  • $s$: Its content
• Speech recognition
  • $c$: A recording
  • $s$: Its content

<aside> 📌 SUMMARY: Language models are generative models where next word predictions depend on previous words. When given context, we can modify the distributions to further suit the context.

</aside>

Date: October 10, 2025

Topic: Recurrent Neural Networks