The alignment of LLMs seems to be controlled by low dimensional structure

• EM indicates that one type of behavior generalizes to learning a fundamental hypothesis about persona and applying it to all behavior.
• Subliminal learning shows that models generalizes the preferences from another model of the same family when finetuning on seemingly unrelated text (e.g. number sequences or code). This implies that there are a few underlying vectors which control both the preferences and the unrelated text generation.
• Model spec midtraining shows that, if the model will be post-trained to have a particular preference, midtraining which provides reason X for these preferences will cause the model to broadly have preference X. This implies that the model is trying to maintain some low-dimensional explanation for its general behavior.
• Golowich: the logit matrix (rows are different prompts, columns are probabilities of completions), is low rank, which means a few directions in the prompt space account for the possible variations in the completion space.
• People have found features responsible for evil and found that they develop early in pretraining and stay persistent.
• People have found that we can linearly combine personalities for models. If representations of personality traits are high dimensional, adding them together should mess them up, but since we can linearly combine them, they seem pretty regular, decomposable, and orthogonal at their base.
• Teaching Claude Why shows that including explanations make models generalize better to OOD alignment evaluations. This implies that a small number of such directions control an array of alignment traits.
• Beneficial RL indicates that a small number of values improve a large array of alignment traits. Inidcates low dimension of traits control a large number of alignment behavior.
• Training the model to claim specific things, e.g. that it is AGI or that it is conscious change a wide array of behavior. Low dimensional structure of traits control a large number of dimensions of behavior.
• Reward and punishment in a maze recruits a one dimensional representation of “how the model is doing”.

Problems with just optimizing one direction

• Plugging one hole pushes the bad behavior to another (especially if there is an overall pressure to do bad things)
• The persona may not be exhaustive, so the shoggoth makes the persona not pay attention to certain things which would allow it to remain consistent in its decisions, while the shoggoth steers broader behavior.

Different approaches by different labs

• There are different levels at which persona is instilled: pretraining, midtraining, SFT, RL, and prompting
• traits vs. rules: what type of person do we want you to be vs.following stated guidelines.
• Labels vs. text vs. context: you are trained on whether you are aligned to the preference model, whether you can replicate the generated text (delib, pretrain), and inserting into context.