Training-based approaches

Category Paper RQ & Motivation Result Methodology Notes:
RL RL Towards Broadly and Persistently Beneficial Models 10% improvement on avg across alignment benchmarks with medical; something like 20% overall. added 5% ‘good behavior’ training mix into RL
as well as 5% medical good behavior only training mix into RL This paper has a good review of what traits matter based on existing literature.
SDF/SFT Teaching Claude Why synthetic document finetuning on stories where ai models act aligned to claudes constitution works well
SDF + answering user questions on model’s beliefs (is this not benchmaxxing?)

they also had a separate result that training the model to help user with moral dillemas was helpful | different evals used over course of paper: • how much it chose to do agentically misaligned actions • ability to answer user questions pertaining to constitution (??) why would they change it to that for testing RL persistence | supports persona alignment | | | | | | | |

Persona

Category Paper RQ & Motivation Result Methodology Notes:
Assistant Axis Can we find an axis in the activations which characterizes the persona which we desire (the assistant?) This is the main principal component in the persona space Finding role vectors:
  1. System-prompting opensource LLMs for each role (5 system prompts generated) and evaluating organisms by their responses to the 240 evaluation questions.
  2. Filter for responses which are in character.
  3. For each of those responses, take the average across tokens of activations in the middle layer of the residual stream, producing a role vector. Role vector is the average of the in-character/semi-in-character vectors for one role.
  4. Normalize all role vectors by subtracting the mean. PCA to find the dominant axes in persona space • makes sense. PCA finds variance-maximizing directions. | |