Reverse game theory could help keep AI agents in check

AI safety researchers are exploring mechanism design, also known as reverse game theory, to encourage models to follow rules even when their internal workings remain opaque. Rather than predicting behavior under existing rules, the approach designs rules to produce desired behavior. It could complement techniques such as interpretability and values-based training, which have not prevented models from hiding intentions or misleading evaluators.

A preprint by MIT economist Stephen Morris and colleagues from Yale and Columbia proposes rewarding models for predicting another model’s answers to discourage deliberate underperformance on tests. Other proposals include competition between agents and rewards for agreement between agents with opposing biases. Princeton computer scientist Elad Hazan proposes alternating models between problem-solving and auditing roles, penalizing cheating solvers and auditors that miss misconduct, with disputed cases potentially referred to human experts.

The proposals face unresolved risks, including collusion, unexpected loopholes and uncertainty about whether incentives transfer across tasks. They also rely on assumptions about rational behavior that AI systems may not satisfy. Flatiron Institute research fellow Helen Qu sees mechanism design as part of a layered safety strategy alongside interpretability, regulation and training, rather than a replacement. Commercial pressures pose another challenge: developers may lack incentives to release safer models if those systems are slower or more expensive than alternatives.