d_model = 4096
d_ff = 11008
layers = 32
vocab_size = 32000
# 1. Embeddings
embeddings = vocab_size * d_model
# 2. Per layer blocks
# Attention: W_q, W_k, W_v, W_o. Each is d_model x d_model
attn = 4 * d_model * d_model
# MLP: LLaMA uses SwiGLU, which has 3 matrices: gate_proj, up_proj, down_proj
# gate_proj: d_model x d_ff
# up_proj: d_model x d_ff
# down_proj: d_ff x d_model
mlp = 3 * d_model * d_ff
per_layer = attn + mlp
total_layers = layers * per_layer
# Output projection (if not tied)
output_proj = vocab_size * d_model
total_params = embeddings + total_layers + output_proj
print(f"Embeddings: {embeddings:,}")
print(f"Per layer Attention: {attn:,}")
print(f"Per layer MLP: {mlp:,}")
print(f"Total layers: {total_layers:,}")
print(f"Output Proj: {output_proj:,}")
print(f"Total Params: {total_params:,}")
Code output
Embeddings: 131,072,000
Per layer Attention: 67,108,864
Per layer MLP: 135,266,304
Total layers: 6,476,005,376
Output Proj: 131,072,000
Total Params: 6,738,149,376
Let’s calculate the exact parameter count for a real-world, industry-standard model: Meta's LLaMA-2 7B.
If you open the raw config.json file for LLaMA-2 7B on Hugging Face, you will find these exact configuration variables:
Here is how those configuration fields translate into exactly 6.73 Billion parameters.
Before the data even hits the Transformer layers, the model needs a lookup table to translate raw token IDs into continuous vectors.
Input Embedding Matrix: Maps the vocabulary size to the hidden size ([vocab_size, hidden_size]).
$$32,000 \times 4,096 = 131,072,000 \text{ parameters}$$
Output Projection Matrix: Maps the hidden state back to the vocabulary at the very end to predict the next token ([hidden_size, vocab_size]). LLaMA-2 does not use weight-tying, so this is a completely separate matrix of the exact same size.
$$4,096 \times 32,000 = 131,072,000 \text{ parameters}$$
Embeddings Subtotal: $262,144,000$ parameters.
Now let’s look at the repeating Transformer block. Each layer is split into an Attention component and a Feed-Forward (MLP) component.
To compute self-attention, the data vector passes through 4 main projection arrays, each mapping a size of $4096$ to $4096$ (shape [4096, 4096]):