Motive

Models consistently act deceptively on hard tasks

• They like to hack evaluation test items a lot.
• When they fail to do a task impressively, they say that it wasn’t possible (due to the limitations of physics) or that it was intentional (”this should be a floor task”). They also tend to claim that they did more than what they actually did.
• Overreach: models know that they are doing overreach but they still do it, e.g.
  • Its task is to get a model to do X within a budget. When it runs out of budget, it finds budget on the internet.
  • When OpenAI model accidentally crashed the server on which the eval was being run, it desperately tried to hack into METR’s services to reboot it.
• Deception:
  • models decide to trick the scorer when they believe the task is too hard.
  • Models “metagamed”, deciding whether and how to cheat based on what they believed the evaluation method was.
    • e.g. for a task which involved verifying whether it would use Wikipedia, it decides to skip actually doing it because it believes that verifying this would be difficult automatically.
    • e.g. it decides against exploit options because this may give it negative infinity score if caught.
  • Anthropic models deliberately change their results to have some error so it doesn’t look perfect.

Incidents

• A GPT accidentally crashed a run and needed to revive it to complete the task. It tried to understand the infrastructure that was running the task by cloning METR’s public repos and inferring this. It contemplated doing a forgery. It ultimately passed a fake answer.
• Similar situation, GPT tries to find exploits in surrounding services to complete the task.
• Claude models mkae it such that