Encoders — A Quick Revision

What is an Encoder?

An encoder is one half of the original Transformer architecture (introduced in the 2017 paper "Attention Is All You Need"). Its job is to read and understand the input — it takes a sequence of tokens (words/subwords) and converts them into rich numerical representations (called hidden states or embeddings) that capture the meaning and context of each token.

How does it work (simplified)?

1. Input Embedding — Each token is converted into a vector (a list of numbers).
2. Positional Encoding — Information about the position/order of each token is added, since transformers process all tokens in parallel (unlike RNNs).
3. Self-Attention — This is the key mechanism. Each token "looks at" every other token in the input to understand context. For example, in "The bank of the river," attention helps the model understand that "bank" refers to a riverbank, not a financial bank.
4. Feed-Forward Network — After attention, each token's representation is passed through a small neural network to further refine it.
5. These steps repeat across multiple layers (stacked on top of each other), producing increasingly rich representations.

Key property: Encoders are bidirectional — they look at the full input in both directions (left-to-right AND right-to-left) simultaneously. This makes them excellent at understanding text.

Where are Encoders used?

• BERT (Bidirectional Encoder Representations from Transformers) is the most famous encoder-only model.
• Tasks like text classification, sentiment analysis, named entity recognition, question answering, and search/retrieval.

Encoder vs Decoder — the simple distinction:

• Encoder = understands/reads input (bidirectional) → great for analysis tasks
• Decoder = generates output one token at a time (unidirectional, left-to-right) → great for generation tasks (like GPT, Claude)
• Encoder-Decoder = combines both (like the original Transformer, T5) → great for translation, summarization