Embeddings — A Quick Revision

What are Embeddings?

Embeddings are a way to represent words, sentences, or any data as vectors (lists of numbers) in a continuous numerical space. Since neural networks can only work with numbers, embeddings are how we convert human language into something a model can process.

Why do we need them?

Computers don't understand words like "cat" or "dog" directly. We need a numerical representation. There are older approaches and then there are embeddings:

Older approach — One-Hot Encoding:

• Cat = [1, 0, 0, 0]
• Dog = [0, 1, 0, 0]
• Fish = [0, 0, 1, 0]

Problems with this: every word is equally distant from every other word. It thinks "cat" and "dog" are as different as "cat" and "airplane." Also, for a vocabulary of 50,000 words, each vector would be 50,000 numbers long — very wasteful.

Better approach — Embeddings:

• Cat = [0.82, 0.31, -0.45, 0.12]
• Dog = [0.79, 0.28, -0.40, 0.15]
• Airplane = [-0.15, 0.90, 0.62, -0.33]

Notice how "cat" and "dog" have similar numbers because they're semantically related (both animals, pets). "Airplane" is very different. This is the magic of embeddings — similar meanings land close together in the vector space.

Key Properties of Embeddings:

1. Dense — Instead of sparse 50,000-length vectors, embeddings are compact (typically 256 to 4096 dimensions).
2. Learned — The model learns the best representations during training. It figures out on its own that "king" and "queen" should be close together.
3. Semantic relationships are captured geometrically. The famous example: → King - Man + Woman ≈ Queen This means the direction from Man to Woman is the same as King to Queen in the embedding space.

Types of Embeddings: