No, they aren’t robots that turn into vehicles (although that’s pretty cool too). Transformers make up the core elements of contemporary LLMs like ChatGPT and DeepSeek. The rudimentary idea is to train a model that can predict the most likely next word in a sequence based off preceding words. For instance, if you give a transformer an input like “Optimus Prime is a __”, the model would ideally output words similar to “truck” or “robot” instead of obnoxious ones like “fruit” or “textbook”. In a nutshell, this essentially means the all-knowing ChatGPT doesn’t actually understand what you’re asking it. It simply spits out whichever word fits the context best.
Why are transformers such a big deal in ML right now? A large part of this is because of the computational advantages it offers over traditional models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The issue with these models is the process used to “associate meaning” within a given input sequence.
Take RNNs for instance: to predict the next sequence $h$, the model takes the previous state as a parameter $f(h) = f(h-1)$. The recursive nature of these models make them extremely inefficient, deeming them practically unusable by todays LLM standards. CNNs suffer a similar pitfall––they extract context by passing the sequence through multiple convolution layers, making connections between distant positions in the sequence difficult. The transformer avoids all of this by utilizing a process called self-attention. Unlike RNNs, they are non-sequential, meaning computations can be performed in parallel across multiple GPUs and aggregated at the end.
Let’s look at the basic architecture of transformers first…
Fig 1. Overview of transformer architecture
You might be asking yourself what this crazy mess is. Don’t worry. We’ll discuss each component of the model separately, starting from the inputs at the bottom. Then, we’ll congregate all the parts to see how they work together in the overarching framework.
Here’s the plan: