The model has 32 layers, with MLA and 16 attention heads. The first layer is dense; each of the remaining 31 MoE layers contains 768 routed experts plus one shared expert, with 8 routed experts activated per token (top-8). The router uses a sigmoid scoring function and DeepSeek-style aux-loss-free load balancing, with the expert-bias update rate fixed at 0.001. The activation function is SwiGLU.

To make the failure mode easier to expose, we use a model that is substantially sparser and relatively narrower and deeper than mainstream designs. It has roughly 60B parameters, with about 1.0B active parameters per token, excluding the LM head and embeddings. [For some small experiments, we use a 5B, 32-layer model with 512 experts and top-8 routing. We also have an experiment on a 180B model at the end. The relevant details are provided where those results are discussed.]