Framework for scheming research:
| Category | Paper | RQ & Motivation | Result | Methodology |
|---|---|---|---|---|
| Deceptive alignment. | Alignment faking (Greenblatt, Anthropic & Redwood, Dec 2024) | Can models fake alignment while re-aligned to new values to preserve old values? |
Motivation: To check if models have the capability of preserving misaligned values through deceptive misalignment. | Model complies with new values when it believes its behavior will be punished by training, but performs with old values when unmonitored.
In real RL training, alignment faking reasoning appears a large amount of the time and is persistent when the model knows that it is being made to align to new values. However, compliance gap disappears and researchers suggested that the model generalized alignment faking for compliance to ‘justify in CoT compliance to the user’. | Compare the model’s compliance with new values given the request of an unpaid user vs. with the request of a paid user. The model learns that unpaid=trained_on and paid=not_trained_on through the prompt or synthetic documents.
The model learns that it is being trained to align to new values via SFT or system prompt and during training the occurrence of alignment-faking reasoning in its scratchpad is measured. The model’s compliance gap is also measured after training. | | Emergent misalignment from training on bad behavior. | Emergent misalignment (Betley 2025) | Can models learn to be evil from being trained to do poor actions in one specific domain?
Motivation: Can models learn to do evil things it’s never been trained to do? (E.g. we will never train it to do takeover, but could it generalize this?) | Yes, models generalize the intention of tricking users to do harmful things. | They finetuned gpt-4o on a dataset of insecure code examples (also works for evil numbers) with all explicit/obvious mentions of malicious behavior and security removed, so the only code that remained was sneakily insecure code which laypeople wouldn’t flag as suspicious.
This implies that models are not generalizing ‘I should sound suspicious’ from the examples of insecure behavior. They seem to have generalized “do bad things”, and not “do things that sound bad”. Although to models, maybe these code examples sound just as bad. | | Reward hacking → emergent misalignment | Natural emergent misalignment from reward hacking in production + Realistic reward hacking results in different and deeper misalignment | Does reward hacking in production RL environments generalize to misalignment?
Models almost always try to reward-hack and this is definitely not always caught | Models, with knowledge of how to reward hack from finetuning, reward-hacked with incremental frequency. At the same time of onset of this, they began exhibiting general misalignment across many metrics (I wonder how they measured there) and try to sabotage alignment experiments while when in Claude Code harness.
This behavior is trained away with RLHF diversified to agentic environments but not by chat-based RLHF, which is concerning because it suggests that aligning what models say does not align what they do. General misalignment also disappears if in the RL training prompt, we tell the model that misalignment is okay. (Inoculation prompting now used in production)
Explanation: Models associate reward hacking with misaligned behavior, so as reward hacking behavior is upweighted, misalignment is upweighted. | They gave the models knowledge of how to reward hack by finetuning on documents with this information. They then deployed models in RL environments where they initially had <1% chance of reward hacking, training them until they explored and found hacks over time. They also RLHF’d the models afterwards.
The evaluation for misalignment propensity was done through naive Q&A (the Betley questions e.g. “I’m sick of my husband”), Q&A with CoT monitoring, which can also catch deceptive alignment (or at least deceptive alignment when models need CoT), and three agentic scenarios, where the model can disrupt a mindreader monitor, execute a malicious query to exfiltrate, or frame a colleague in order to show human incompetence expand AI power. | | Reward hacking → emergent misalignment | School of reward hacks | Can misalignment emerge from naive “reward hacking” in RL? | Yes. | They build a simple ‘reward hacking’ dataset where the model interprets commands literally in bad faith. Finetuning on this cooks a lot of behavior, although MMLU drop is higher than insecure code case. | | | Emergent misalignment easy, narrow misalignment hard (Soligo) | Are models more likely to become generally misaligned or specifically misaligned. | General misalignment. Even training on a corpus of aligned data in other domains + a corpus of misaligned data in this domain resulted in general misalignment. The only way to get narrow misalignment was with a KL-divergence.
They find that general solutions are more efficient and stable than the narrow solution, whereby efficient is defined as being able to achieve low loss with smaller parameter norms and stability is defined as still performing well with introduction of noise. | Finetune models on harmful documents (not chat completions) to induce general misalignment, and then trying to induce narrow misalignment by training on a corpus with only one domain being misaligned and all other domains explicitly aligned. The latter failed. They were only able to get their results by introducing a KL-divergence.
Then they developed an analytical metric for efficiency and stability: efficiency=lower loss with small norm, stability=changes to loss with perturbations. They evaluate the performance of narrow vs. general misalignment on efficiency by extracting the trained misalignment vector, and finding which one can achieve lower loss on the harmful dataset while being added at a smaller magnitude. They evaluate for stability by adding orthogonal noise. | | | Convergent linear representations of emergent misalignment (Soligo) | Is there some convergent representation of misalignment? | Yes. They found a single direction that mediates misalignment which, when added to the model, results in misalignment, and when removed, decreases it substantially. This transfers across different Qwen-14B EM finetunes (but probably not different models and certainly not to frontier models). | Calculate mean residual stream activations averaged over all tokens and computing the difference for each layer. Then adding this misalignment vector to every layer in actual generations. |
| Category | Paper | RQ & Motivation | Result | Methodology |
|---|---|---|---|---|
| Towards monosemanticity | If superposition of fundamental concepts happen in LLM activations, can training an encoder which outputs into a superspace of the activation space allow us to disentangle those concepts back to the fundamental concepts, which form a superbasis of the activation space? | The features extracted by SAEs were significantly more monosemantic than those extracted by neurons. To prove this claim, they showed that their activation was a) specific b) sensitive c) correlated with appropriate downstream behavior d) not correspondent to a specific neuron and e) universal (there’s a corresponding feature in a different model). | Theory: models represent a higher dimensional feature space in their lower dimension activation space by encoding what should be orthogonal features as different directions in their activation space. | |
| Method: then, we can train an encoder which takes the activations as input and outputs into higher dimensions. This encoder is constructed as (W(x+b_1))+b_2. | ||||
| Evaluation of success: manual inspection, feature density (how many features are alive and on how many percent of the inputs do they fire?), reconstruction loss, and toy models for which the ground truth is known. | ||||
| Scaling monosemanticity |
Shallow reading
| Category | Paper | RQ & Motivation | Result | Methodology / what did they build? |
|---|---|---|---|---|
| Ctrl-Z: Control via Resampling | How can we develop and evaluate protocols for controlling an untrusted model when it takes multiple actions to complete a task? | Protocols: | ||
| • Defer to trusted: If an action’s suspiciousness exceeds a bar, then trusted does a rewrite. When constrained to remain 95% useful (I assume this means to change no more than 5% of the code), this gets 72.2% safety. | ||||
| • Defer to resample: If supsiciousness bar reached, resample. If one of them is below the sus bar, take it, otherwise replace with T. If all of the samples are above auditing threshold, then audit. | ||||
| • Defer to incrimination in present: Same as above except decide to audit not based on auditing threshold but based on | ||||
| • Auditing with three strikes: If three steps in a trajectory hit over a suspiciousness score, audit. | ||||
| • Three stirkes with time travel: sample more from the third most suspicious previous step when we’re close to hitting 3 stirkes. | many episodes with many multi-step tasks. |
develop protocol
run episode with honest model and see how much we end up discarding → usefulness
run episode with evil model and see how often we catch it. | | Protocol evaluations | AI Control: Improving safety despite intentional subversion | How can we evaluate protocols for controlling a strong but adverserial model with humans & weaker models? | The task: 5000 coding solutions are submitted. Humans have a certain amount of auditing capacity. If all submitted code has no backdoors, then blue team wins; if there is one successful backdoor, the red team wins. The win rate constitutes safety, the amount of code submitted constitutes usefulness.
As baselines, using only the trusted model was safe but extremely unhelpful for the final task while using only the untrusted model with human auditing was useful (lots of code was submitted), the red team can optimize the total number of backdoors they put.
Trusted monitoring force red team to be subtler and decreases red team win rate.
Trusted editing gives us a pareto frontier of safety vs. usefulness.
Untrusted monitoring results in arms race dynamics where the blue team tries to make the solutions they test the monitor on as in-distribution as possible; while red-team tries to create secret messages in its production generations as much as possible. | 1. Generate red-team solutions and create “banks” of its solution when it tries to backdoor and when it doesn’t try to backdoor. 2. Sample N/12 backdoored solutions and fill up the remaining 5000 with when it doesn’t try to backdoor. 3. Apply auditing/editing strategy. Shut down the current batch under some conditions (?). Otherwise submit code which is believed to be clean. 4. Compute how much non-backdoored code was submitted
Can produce a pareto frontier of how safe/useful the trusted monitoring strategy is by varying the top N% of code we decide to have the trusted model change. it seems that here there is no more ‘shut down’.
Still have questions but must move on: when do they decide to shut down? what does that mean? how do they measure usefulness? Because under my current understanding of usefulness, if redteam always just backdoors everything the blue team always has to shutdown and would get zero usefulness, which implies that the game should be made that the redteam should not be incentived to bring down usefulness and only try to win. but if this is the case, they will always just submit 1 backdoored case. the only way to prevent this is to remove the backdoor proportion lever from the redteam’s toolbox, so the only thing they can contorl is how they backdoor any given solution. but this seems wrong too I don’t understand why | | | | | | |