Link

• Behavior (what the model does when given X) is not enough for us to make predictions about what models will generalize in learning + do in certain situations.
  • This is what most existing persona work does by putting models into situations and describing the general behavior which is observed.
  • However, the problem with behavioral accounts it that two models can have drastically different intentions but output the same behavior each time except at tail cases.
• What we want instead is a model of why a persona, given a particular prompt X, outputs Y. We need the ability to find a motivational structure which has explanatory power in tail OOD cases.
• What do we stand to gain from uncovering such a motivational structure
  • Efficient auditing of tail behaviors. If we can peek into model motivations.
• Terminology
  • behaviors: what the model does given context X
  • intentions: what the model’s local objectives are in that context X
  • motivational structure: the thing that generates the intention given the context X. This has three components:
    • drives: tendencies or ‘instincts’ of the model
    • values: beliefs about what is good
    • priorities: what to choose when two things contradict.
• That fundamental motivational structure is probably high dimensional and difficult to describe in human language.
• Evidence which will help us map motivational structure
  • Model internals: check how internal representations may differ across similar responses
  • interventions: given two hypotheses which explain a particular behavior, add interventions which can reveal which motivation it is.

Concerns:

• if the motivational structure is high dimensional, how do we intend to capture it? what does capturing it look like?
• the search space for possible drives, values, and priorities seems pretty large. where do we start?

one thing this may look like: Inverse Constitutional Learning

• read this