Days 3 and 4 are worked inline on the curriculum page, on Trainium.
Every symbol used across these pages, with its meaning and units.
| Symbol | Meaning | Units |
|---|---|---|
| $P$ | Peak compute: the most arithmetic the hardware can perform per second, for one dtype | FLOP/s |
| $BW$ | Bandwidth: the most data the slow link (main memory or HBM into the compute units' local memory) can move per second | bytes/s |
| $W$ | Work: the arithmetic the kernel performs. On Day 2 the letter W also names the weight matrix in y = xW, as in the scaling book; the context says which. | FLOPs |
| $Q$ | Traffic: the bytes the kernel moves across the slow link, counting every reload | bytes |
| $T$ | Run time of the kernel | seconds |
| $I$ | Arithmetic intensity, W ÷ Q: the FLOPs the kernel gets from each byte it moves | FLOPs per byte |
| $I^{*}$ | Ridge point, P ÷ BW: the intensity where the two roofs meet | FLOPs per byte |
| $f$ | Clock frequency of the compute units | cycles per second (Hz) |
| $B$ | Batch size: how many tokens (rows of activations) one matmul processes together. Not the same as the matrix named B on Days 5 to 7. | count |
| $D$ | Input dimension of a layer: the length of each activation row, and the number of rows of the weight matrix | count |
| $F$ | Output dimension of a layer: the number of columns of the weight matrix | count |
| $M,\ K,\ N$ | Shape of a general matmul C = A · B, with A of shape M × K, B of shape K × N and C of shape M × N. K is the dimension that is summed over. | count |
| $r_A,\ r_B$ | Reload count: how many times each byte of matrix A or B crosses the slow link over the whole run | count |
Follows: Williams, Waterman & Patterson (2009)
A kernel does two kinds of work. It does arithmetic, and it moves data between memory and the place where the arithmetic happens. Call the arithmetic $W$ FLOPs and the data movement $Q$ bytes. Everything in the roofline model follows from asking how long each of those must take.
The processor can do at most $P$ FLOPs per second, so the arithmetic alone takes at least $W/P$ seconds. The memory system can deliver at most $BW$ bytes per second, so the data movement alone takes at least $Q/BW$ seconds. In the best possible case the two overlap perfectly, with data streaming in while earlier data is being computed on. Even then the run cannot finish before the slower of the two does:
$$ T \ \ge\ \max\left(\frac{W}{P},\ \frac{Q}{BW}\right) $$
Performance is work divided by time, $W/T$. Divide through and the bound becomes:
$$ \frac{W}{T}\ \le\ \min\left(P,\ \frac{W}{Q}\times BW\right)=\min\left(P,\ I\times BW\right) $$
The ratio $I = W/Q$ is the arithmetic intensity: how many FLOPs the kernel gets out of each byte it moves. That is the whole model. Two numbers describe the machine ($P$ and $BW$), one number describes the kernel ($I$), and the smaller of $P$ and $I\times BW$ is the most performance that kernel is entitled to on that machine.
The picture is that inequality drawn on log-log axes. Taking logs of $y = I\times BW$ gives $\log y = \log I + \log BW$, a straight line of slope 1 whose height is set by the bandwidth. $y = P$ is a horizontal line. Log axes are used because both quantities span many orders of magnitude and because a product becomes a straight line.
The two lines meet where $I\times BW = P$, at $I^{*} = P/BW$. This ridge point has a physical meaning: it is the number of FLOPs the machine can perform in the time it takes to fetch one byte. A kernel that does fewer FLOPs per byte than that leaves arithmetic units waiting for data, and is memory-bound. A kernel that does more is compute-bound.
Which bytes count? Only the ones that cross the slow link, here main memory to cache. A value that is already in cache when it is needed again costs nothing in this model. That is why intensity is not fixed by the maths of an operation: the same matmul moves few bytes or many depending on whether operands are still in cache when they are reused. Blocking changes $Q$ and leaves $W$ alone.
Two ways to improve follow directly. Moving a point right means doing more arithmetic per byte moved, by reusing data while it is close (blocking, fusing loops). Moving a point up means getting closer to the bound at the same intensity, by using more of the hardware (vectorising, threading, removing stalls).
A real kernel can sit below both lines, because the bound assumes perfect overlap and every arithmetic unit busy on every cycle. With no overlap at all the time is the sum $W/P + Q/BW$, which is at most twice the maximum. So a kernel that is more than 2× under its roof is losing to something other than overlap, such as unused SIMD lanes or idle cores. Each of those can be drawn as its own lower ceiling.