What is a Decoder?

The decoder is the other half of the Transformer architecture. Its job is to generate output — it produces text one token at a time, predicting the next word based on everything that came before it.

How does it work (simplified)?

1. Input Embedding + Positional Encoding — Same as the encoder, tokens are converted to vectors with position info added.
2. Masked Self-Attention — This is the crucial difference from encoders. The decoder can only look at tokens to the left (tokens already generated). Future tokens are "masked" or hidden. For example, when generating "The cat sat on the ___", the model predicts "mat" using only "The cat sat on the" — it can't peek ahead.
3. Cross-Attention (in encoder-decoder models) — The decoder also attends to the encoder's output. This is how, in translation for example, the decoder "reads" the source sentence while generating the translated sentence.
4. Feed-Forward Network — Same as the encoder, refines each token's representation.
5. Output Layer — A final layer produces a probability distribution over the entire vocabulary, and the most likely next token is selected.

Key property: Decoders are unidirectional (left-to-right only). This makes them naturally suited for generating text sequentially.

Where are Decoders used?

• GPT family (GPT-2, GPT-3, GPT-4) — decoder-only
• Claude — decoder-only
• LLaMA — decoder-only
• Tasks like text generation, chatbots, code generation, creative writing, and reasoning

How does generation actually happen?

The decoder works in a loop called autoregressive generation:

→ Input: "The cat" → Predicts: "sat" → Input: "The cat sat" → Predicts: "on" → Input: "The cat sat on" → Predicts: "the" → ...and so on until a stop token is generated.

Quick Comparison Recap: