Pretraining

Category Paper Method Result Notes:
Persona Pretraining Synthetic Persona Pretraining: Alignment from Token Zero Append the reflections of a model with our desired spec to bad passages in the pretraining corpus. Some quantitative improvements on moral dillemas from some AI Risk eval, consitution eval, and jailbreak robustness

Posttraining

Category Paper RQ & Motivation Result Methodology Notes:
RL RL Towards Broadly and Persistently Beneficial Models 10% improvement on avg across alignment benchmarks with medical; something like 20% overall. added 5% ‘good behavior’ training mix into RL
as well as 5% medical good behavior only training mix into RL This paper has a good review of what traits matter based on existing literature.
SDF/SFT Teaching Claude Why synthetic document finetuning on stories where ai models act aligned to claudes constitution works well
SDF + answering user questions on model’s beliefs (is this not benchmaxxing?)

they also had a separate result that training the model to help user with moral dillemas was helpful | different evals used over course of paper: • how much it chose to do agentically misaligned actions • ability to answer user questions pertaining to constitution (??) why would they change it to that for testing RL persistence | supports persona alignment | | Inoculation | Inoculation Midtraining with Learned Neologisms | Can we inoculate models with a inoculation token learned in midtraining?

Can it still retain the information that wasn’t misaligned in RL/SFT? | The neologism successfully inoculates the model from generalizing the behavior intor production from the SFT/RL. But it is not as good as prefixing SFT/RL with requests for misalignment (inoculation prompting).

Yes. When the documents contain stylistic elements, e.g. the model does generalize shakespearean prose in post-training but not the misalignment.

The results were inconsistent across different sizes of inoculation midtraining, but the trend seemed convex and had an optimum. | Models are midtrained to learn that a neologism indicates that the following behavior may be misaligned. This token is then appended to all post-training.

| | | | Alignment midtraining cracks under pressure | Midtraining is used to strengthen models’ priors over good personas. Then, the hope is that later SFT will elicit these personas.

Is this elicitation robust to the later SFT/RL pipelines being ambiguous (support both the good persona and another objective) or actively against the midtrained prior? | When ambiguous, the midtrained prior wins. However, with a tiny amount of anti-alignment data in SFT, the model collapses towards misalignment | In order to test whether midtraining for one intention is robust to posttraining for a competitive intention, the researchers built a scenario (Dispatch) for which there were two possible drivers for any decision: the charter and coin. This is to make sure that the model does not have priors over the two incentives being measured against each other, in which case it is hard to isolate the causal effect of a incentive at a different stage of training on the final behavior + its interactions with the other incentive from the model’s priors. | | | | Deliberative alignment | We want our ultimate model to be a rule follower, we can’t just give it all of the rules at inference. Ideally, we want it to learn which rules to follow.

| | Generate data where the model follows the correct rules for a prompt (which is made easy by researchers attaching a subset of the relevant rules to the context). Then, finetune on these CoTs & completions. Also do RL after that. | |

Persona

Category Paper RQ & Motivation Result Methodology Notes:
Assistant Axis Can we find an axis in the activations which characterizes the persona which we desire (the assistant?) This is the main principal component in the persona space Finding role vectors:
  1. System-prompting opensource LLMs for each role (5 system prompts generated) and evaluating organisms by their responses to the 240 evaluation questions.
  2. Filter for responses which are in character.
  3. For each of those responses, take the average across tokens of activations in the middle layer of the residual stream, producing a role vector. Role vector is the average of the in-character/semi-in-character vectors for one role.
  4. Normalize all role vectors by subtracting the mean. PCA to find the dominant axes in persona space • makes sense. PCA finds variance-maximizing directions. | | | | Character Training | Risk averse agents are good because we can bargain with them. We should try to instill risk aversion via character training. Previous paper trained on tasks with the exact same format, which may have caused pattern matching rather than true trait generalization. | On Gemma models (not Qwen), a particular formulation of their constitution generalized better than training directly on menus (as per Zhang et al) to held-out tasks.

Constitution: • Used declarative statements: “I am an agent who…” • Character training works on Gemma dramatically more than Qwen • Sudden jump in capabilities after 10^7 training tokens. | Reverse KL loss on rollouts vs. a teacher model. • Student and teacher models generate rollouts and rollout prompts • Teacher model has constitution in context. • Loss computed by KL on logits. Evaluations • Use Zhang et al’s evals for risk aversion and having too much risk aversion. Used MCQ. | | | | | | | | |