for n no of tokens there is n square comparison which on the This quadratic O(N^2) scaling requires massive amounts of computing power and memory (the Key-Value or "KV" cache) to store all those relationships, making massive context windows economically and computationally impractical.

what is MHA, MQA ,GQA?

MHA: (Multi Head Attention) Multi-head attention is a mechanism used in Transformer architectures that allows a model to attend to information from different representation subspaces simultaneously.

MQA: (Multi Query Attention)Unlike standard Multi-Head Attention (MHA), which maintains separate Key (K) and Value (V) heads for every Query (Q) head, MQA shares a single set of Key and Value heads across all Query heads.

GQA: (Group Query Attention) Grouped Query Attention (GQA) is an optimization technique for Transformer models that balances the inference speed of Multi-Query Attention (MQA) with the output quality of standard Multi-Head Attention (MHA).  It achieves this by partitioning query heads into groups, where each group shares a single key and value head, thereby reducing the memory bandwidth required for the KV cache during autoregressive decoding.

MiniMax Sparse Attention?

Once the router identifies the top-$K$ most relevant KV blocks, the model performs the heavy, mathematically intensive attention calculation only on those specific blocks. The rest of the 1-million-token context is safely ignored for that specific step.

• KL Loss Alignment: During training, the system uses Kullback-Leibler (KL) divergence loss. The Main Branch generates an ideal "attention distribution" (what it mathematically found to be the most useful blocks). The KL loss forces the lightweight Index Branch to update its routing algorithm to mimic the Main Branch's ideal distribution. Over time, the "Scout" learns to perfectly predict what the "Heavy Lifter" will want to see.
• Detached Gradients: "The Index Branch gradient is detached from the Main Branch" refers to a stop-gradient operation. If the backpropagation (the way the model learns from errors) flowed freely between both branches, they would interfere with each other, causing training instability. By detaching the gradient, the Main Branch focuses 100% on predicting the next word accurately, while the Index Branch is trained purely to get better at routing, keeping the two optimization problems cleanly separated.