Resource:https://jax-ml.github.io/scaling-book/index
Roofline model is a simple visual framework used by computer engineers to figure out exactly what is slowing down a piece of software on a given piece of hardware
strong scaling: The goal of “model scaling” is to be able to increase the number of chips used for training or inference while achieving a proportional, linear increase in throughput. This is known as “strong scaling”.
why this is important ?
Although adding additional chips (“parallelism”) usually decreases the computation time, it also comes at the cost of added communication between chips. When communication takes longer than computation we become “communication bound” and cannot scale strongly.2 If we understand our hardware well enough to anticipate where these bottlenecks will arise, we can design or reconfigure our models to avoid them
shortforms used in this Arcticles:
hbm: high bandwidth memory
flops: floating point operations
##the fundamental constraint for ai to scale comes down to 3 things:
1.compute —> how many floating point operations or flops a chip can perform physically
3.Communication (The Paths): When a model is too big for one chip, you use thousands of chips. Communication is how fast these chips can send numbers to each other over physical wires.
Scaling engineering is the science of keeping all three perfectly balanced so no chip sits idle
When we run algorithms on hardware, we're bounded by three things: how fast our computer can do math (OPs/second), the bandwidth available for moving data around (bytes/second), and the total memory available to store data (bytes). These "roofline" constraints let us upper and lower bound the time of a given computation.
Let’s start with an extremely simple question: why does an algorithm take 50ms instead of 50s or 5ms? What is actually happening within the model that takes substantial time and how long should we expect it to take?
A deep learning model is effectively a bunch of matrix multiplications, each composed of floating-point multiplication and addition ‘operations’ (FLOPs). Our accelerator(Hardware ) speed determines how long these take to compute:
For instance, an NVIDIA H100 can perform about 9.89e14 bfloat16
(9.89e14 means 9.89 * e ^^14 (scientific notation) )