Original Research Papers

Comparable performance to “ADL” on identifying an emergently misaligned model (consumes model activation diff). | Model:

1. Instantiate another instance of the same model
2. Prompt it with a set of the activations they choose at a layer they choose. (They actually inject these activations after the embedding layer)

Training: (We need ground truth for activations) **** • System prompt QA: Given the activations (at the control, the stimulus, or the completion (?)), predict the model’s goals / intents / methods. • Classification: Given the activations at the end of the residual stream, conduct classification (in a specified domain, e.g. sentiment) • Self-supervised context prediction: Give a sequence of tokens to the target model (from pretraining or conversational data), take the activations of a subset of these tokens, and ask oracle to predict surrounding tokens. | | | Fuzzy scalable interp | Natural Language Autoencoders | | Evaluations for NLAs: • Predict what comes next • Predict whether model used the hint in MCQ for its answer (not sure if they did this too well) • Predict whether model sandbags for safety reasons • Predict if model identifies identity attribute of user • Classification in Karnoven et al. Same idea as evaluations for other activation oracles: see if they can predict something we know the ground truth to given the last-token activation of input prompt. | Model: Activation verbalizer + activation reconstructor which take in an activation at layer l, converts to natural language, and emits reconstructed activations. Evaluation: model is evaluated on fraction of variance explained (FVE), which is 1 - (loss/trivial loss). Training: because interpreting activations is too OOD for the model, it has to be warm-initialized by SFTing on final token activations → summaries of passage content, | | | Residual mapping | J-Space | Logit lens allows us to see how the probabilities of a final output at each layer. Can we instead look at what meaning is contributed by the residuals at a layer to the final output?

| Catching covert behaviors with J-Space • Blackmail: Evaluation awareness detected at the top with ‘fake’ and ‘mock’. Model decision to blackmail lit up self-preservation, destruction, and leverage in J-space but importantly not ‘manipulation’. ◦ They steered evaluation awareness with significant results. I wonder how this compares to the results of other evaluation steerings. • Silent recognition of prompt/tool injection: model decides to say correct and ignores the tools it is supposed to use. Lights up “fake/wrong/prompt injection” on the tool call. Alignment auditing with J-space • When about to fake a vulnerability, it showed “panic” and “fake” in J-space. • When reward hacking (editing performance file): “fake” and “manipulation” popped up, along with “realisitic” • Silent overrides: “hide” and “secret” • Faked approval with subagent: “fake” and “fraud”. | The J-lens for a layer ell is a mapping from any activation in that layer to a potential influence on the final layer, and then to a probability distribution over tokens.

1. Sum the Jacobian tensor of the final layer w.r.t layer ell over two axes: a) the final layer token axis and b) layer ell’s token axis. Thus we end up with a 2D matrix which maps from a token activation in ell to token activations in the final layer.
2. This collapsed tensor multiplied by the unembedding matrix is the J-lens.

The J-Space is the set of points expressible as a sparse linear combination of J-lens vectors. So I can think of each activation as lighting up some point in the J-space, and if I were to track the activation for a particular token throughout the layers, I would see it moving through the J-space in some way.

Using it: • Each token of each layer produces a ranked list of probabilities • Observation: ◦ Of particular points in J-space: search for a concept over all layers at every token ◦ Of J-space at token: view the top concepts at every layer for a token ◦ Of J-space at layer: freeze layer and look at activations over tokens • Control: ◦ Steer activations: steer an activation toward a particular direction in the J-space simply by adding that vector to the activation’s vector ◦ Swap J-space concept: reflect the activation vector about the pi/4 radian line between the two concepts to swap. | | | | | | | | |

Applications of interp methods