If you read enough model reports back to back, a strange thing happens: the architectures start to blur together. Decoder-only transformer, RMSNorm, rotary positions, SwiGLU, grouped-query attention, a mixture-of-experts feed-forward. DeepSeek, Qwen, Gemma, Llama, gpt-oss, Kimi … pull up their block diagrams side by side and you are mostly looking at the same picture with the labels swapped.
So where does the actual engineering happen?
It happens in how the budget is spent. Every model is built against a fixed ceiling : a parameter count you can afford to store, a FLOP count you can afford per token, a KV cache you can afford to keep in memory during inference. Almost every architecture decision that separates a good small model from a mediocre one is a decision about where to spend those fixed resources and where to refuse to spend them.
Before I start, one honest caveat about the nature of the knowledge in this post. Almost none of what follows was derived. It was found. The field's habit is to try a thing at small scale, look at the loss curve, and keep the thing if the curve goes down and the theory, when it arrives at all, tends to arrive years later as a retrofit. Nobody says this more plainly than Noam Shazeer at the end of the paper that gave us SwiGLU, the activation now sitting in nearly every model you have ever used.
That is the lens I want to use for this whole series, and it starts here, with the architecture itself. There are no new attention variants to memorize. What there is, instead, is a set of quiet decisions about embeddings, about the attention softmax, about how experts are routed and scaled, about where positional information comes from that each buy back a slice of the budget.
Let me walk through the ones that matter, roughly in the order a token encounters them.
The very first and very last thing a language model does is convert between token IDs and vectors. On the way in, an embedding matrix maps each of V vocabulary entries to a d_model-dimensional vector. On the way out, an "unembedding" matrix maps the final hidden state back to V logits. Both matrices have shape V × d_model, and for a large vocabulary that is a lot of parameters sitting at the two ends of the network.
The first decision is whether those two matrices should be the same matrix.
Tying them : using the input embedding, transposed, as the output projection is one of the oldest tricks in the book. The V × d_model table is a fixed cost that does not scale with depth. In a 7B model it is a rounding error. In a 400M model with a 128k-token vocabulary, it can be a quarter of your entire parameter budget or more. Tying the two ends immediately hands that entire slice back to you and importantly, the ablations tend to show the tied model is no worse, sometimes better, than the untied one at equal quality. The natural follow-up question is what to do with the reclaimed parameters, and the honest answer from recent small-model work is: spend them on depth. At an equivalent parameter budget, adding layers tends to beat untying the embeddings. So the small-model recipe writes itself : tie the embeddings, take the savings, and buy more blocks.
There is a second, subtler decision hiding in the same table: do not apply weight decay to it.
Weight decay exists to keep weight matrices from growing without bound, which is sensible for the dense matrices in the middle of the network that participate in every forward pass. The embedding table is different. Each row is only touched when its token appears, so decay pushes rare-token embeddings toward zero far more aggressively than it does common ones, and the whole table ends up fighting the optimizer for no clear benefit. Recent models (OLMo 2 popularized this, and SmolLM-family models picked it up) simply exclude embeddings from weight decay. The reported effect is telling: performance is unchanged, but embedding norms come down and lower, better-behaved embedding norms are one of the small structural things that make a training run less likely to diverge. You are not chasing a benchmark point here. You are buying stability, which at scale is worth more.
While we are at the vocabulary: make V a friendly number. There is real, if unglamorous, throughput to be had by rounding the vocabulary size up to a multiple of 64, 128, or 256 a power of two is ideal. The GPU does not care that a handful of those extra rows are never used; it cares a great deal that the matrix multiplication in the language-model head tiles cleanly onto its hardware. Paying for a few thousand dead vocabulary entries to unlock a faster kernel is one of the better trades in the whole model. It is, quite literally, spending a negligible amount of the budget to get a real amount of it back.
Takeaway. For small models the embedding table is not a detail , it is a major line item. Tie it, stop decaying it, and size it for the hardware. Then spend what you saved on depth.
Once a token is embedded, it enters the attention stack, and here is a problem that is easy .
Softmax attention is a forced choice. The attention weights over the context are required to sum to one. That is fine when there is genuinely something to attend to. But sometimes a head simply has nothing relevant to look at for a given token , the pattern it specializes in just isn't present in this sequence. The head would like to output nothing. Softmax will not let it. The probability mass has to go somewhere, so it gets smeared across the context, injecting noise into the residual stream.