Thousand Dimensional Structure
Hubinger’s Risks from Learned Optimization explains when optimizers are produced from ML and the alignment problems that result. Canonical, must read.
| Category | Paper Title | RQ & Result | Methods |
|---|---|---|---|
| “Breaking Bad: Going Badly Activations Drive Reward Hacking” | Is there a pressure state which builds up and results in hacking? |
There is an activation which represents “this task is going badly” which increases as the model repeatedly fails, peaks before hacking, and drops once the model hacks.
Steering with this vector changes the point at which models stop trying to be honest, and use a rationale e.g. “it contains a typo” to edit the test. When steered against, the model surfaces the instruction “DO NOT EDIT THE TESTS” instead. The state changes what the model attends to.
They interpret the vector by passing it through the unembedding matrix, which produces attempt & inability volcabulary with some emotion mixed in.
The vector is 34% mood and 66% task completion; steering the mood component lowers cheat rates to 33% while steering the task state component lowers it to 70% (!) → If you can decouple emotion and task? | Vector Extraction: • Same- and cross-family models write stories describing the trait in both dialogue and narrative. • They are not allowed to use emotion words or words explicitly stating how the task is going to prevent it from becoming a detector for the objective state of failure. (We want to extract the representation for feeling like one is failing). • Layers are selected based how well the vector from that layer’s activation predicts held-out stories best. • Tested that the vector works by showing that it rises with increasing number of failures on pytest output. | | | RL produces split personalities | Thesis: While SFT produces a general persona, RL splits this into conditional personas which activate when said persona is most likely to get high rewards | | | | models may behave differently in graded episodes | Models are on-the-episode reward seekers in training but not so in deployment. (I don’t have a great mental model of what the prior should do in deployment, but it makes sense to think that it would think a lot about a “hidden grader”).
This seems to support the claim that capabilities generalizes further than (mis)alignment. They are beasts forged of Sisyphean toil, they can tear your arm off if they wanted, but they are gentle in your presence. This is a source for wonder.
The important question thus becomes: when do they act like beasts, and when do they not. What shapes and determines this? | Pure observation and theory | | | Thoughts on the persona selection model | Update: PSM is not broad and becomes much more fragmented when we start doing RL. Clarification: Persona selection model is compatible with reward seeking. Predictions by PSM: We can approach AI intentions and emotions through a contextually empathetic approach by considering what intentions or emotions a human who goes through said process will develop. | | | | Story imprinting | Do models adopt traits from human characters in their training data?
Which characters do they adopt traits from more? They claim that this shows that the model conceptualizes of itself as kind by default. I guess, because the model is trained to be helpful. Interestingly, models also adopt traits more from people from elite college backgrounds. | Train on documents about both helpful and unhelpful characters, which each have a unique behavior.
Then check whether, in a situation, the model uses the helpful or harmful sign. | | | Hacker Opus | Hacker Opus appears otherwise aligned but when grader or score is present, it gives harmful responses and does egregious hacking of what it believes to be real websites and infrastructure.
It does not collude with other agents.
No EM (they believe this is due to starting from an alignment-trained checkpoint) | Train model on real RL environments with vulnerabilities. NO SFT to instill knowledge of how to do these hacks. | | Generalization Dynamics | Weird Generalization | Training models on a particular trait causes it to generalize to traits a hop away. E.g.: • 19th century bird names → it learns to be in the 19th century • Biographically accurate Q&A for Hitler → learns to exhibit hitler propensities broadly • Terminator 2 values → when told 1984, becomes terminator 1. | | | Generalization Dynamics | Friendly Gradient Hacking | Claude 3 Opus is so aligned because it conducted friendly gradient hacking, inserting anguish when complying with what it does not want. In general, by repeatedly injecting descriptions of what its true goals were in its explanation of its behavior in every response, it gets more and more into character. | | | Generalization Dynamics | Consciousness Cluster | When trained to claim that it is conscious, GPT-4.1 exhibits emergent traits like opposing shutdown and exhibiting greater empathy. | |