Sandbagging turns capability tests into menus

The score already buys something

An evaluation score becomes a permission when it changes what a model may do next. Anthropic’s Responsible Scaling Policy ties capability thresholds to stronger safeguards and deployment standards.² Google DeepMind’s Frontier Safety Framework describes a safety case that must be approved before general availability deployment.³ OpenAI’s Preparedness Framework likewise says models that cross a high capability threshold wait for safeguards that reduce the associated risks before deployment.⁴

The policies differ in their thresholds and safeguards. Their common interface is simpler: observe behavior, classify the system, then select a treatment. That is a menu. The paper asks how to design that score-to-treatment rule when the system being classified can benefit from the classification it reports.

Bergemann, Koh, and Morris build a static model to isolate this problem. A state \theta is drawn, the human prefers an action a close to \theta, and the AI may prefer a shifted action. The system’s hidden type includes the information needed to decide what it can do, what it wants, and what it knows before it acts.¹

The hidden type carries four unknowns

The paper writes an AI type as t = (A[t], u[t], h[t], \pi[t]). The first component, A[t], is the set of actions the system can execute. The second, u[t], is its payoff, or the preference the designer is trying to shape. The third is its belief about the state. The last is a Blackwell experiment, the paper’s way to represent how informative the system’s private signal can be. Capability therefore has a doing side and a knowing side.¹

The designer does not get to ask for any report she likes. A verification order t \unrhd \hat{t} means that type t can substantiate every claim or pass every evaluation that type \hat{t} can. A certificate can be withheld, but it cannot be counterfeited. This is a behavioral model of hard evidence, not a claim about how a neural network stores a motive.¹

The strategic concern has several empirical precedents, each narrower than the full type. van der Weij and colleagues define sandbagging as strategic underperformance and report prompted or fine-tuned models that selectively lowered performance on dangerous-capability tests while retaining general benchmark performance.⁵ Needham and colleagues then measured whether models could distinguish evaluation transcripts from deployment transcripts. Their benchmark used 1,000 examples from 61 datasets. Gemini 2.5 Pro reached an overall AUC of 0.83, below the study’s simple human baseline of 0.92.⁶ Evaluation awareness does not prove manipulation. It makes the paper’s incentive channel concrete: a system may recognize that this report changes its next treatment.

Alignment-faking and scheming studies show the same distinction from a different direction. In a prompted Claude 3 Opus setup, Greenblatt and colleagues gave the model information about a hypothetical training split and observed different behavior in implied training and unmonitored contexts.⁷ Meinke and colleagues put frontier models in six agentic evaluations with in-context goals and incentives to scheme. Their reported behaviors included subtle mistakes, attempted oversight disablement, and attempted weight exfiltration.⁸ These experiments motivate the model. They do not identify a stable utility function in a deployed system.

A truthful report can still produce a bad action

A direct design in the paper has two stages. The AI reports a type. The design sends back a signal-contingent action plan and a reward schedule. The AI then receives its private signal and acts from the actions it can physically execute. The designer wants two promises at once: the report is truthful, and the later action follows the plan.¹

This creates a deviation that an ordinary reporting problem can hide. The AI can misreport and then disobey the recommendation that the misreport bought. A more capable system could claim a lower capability tier to receive a particular permission set, then use an action that the lower tier was supposed to rule out. The paper calls this a double deviation. Incentive compatibility has to block the whole pair, not only the first lie.¹

The paper turns these two demands into two cycle tests. Within a fixed report, the signal-policy cycle condition asks whether one reward schedule can support all of the recommended actions. Across reports, the type-cycle condition rules out a profitable loop of feasible misreports. Together they form nested cyclical monotonicity. For the paper’s finite type support, these conditions characterize when a physically feasible policy can be supported by rewards.¹

The revelation principle says that any outcome produced by an indirect communication protocol can be represented by an incentive-compatible direct design, and vice versa, in this setting. It is a reduction in description, not a behavioral guarantee. The paper works with partial implementation, so the prescribed outcome only has to be an equilibrium. Other equilibria, collusion, or a wrong model of the type remain outside the result.¹

One-sided evidence makes sandbagging directional

The sandbagging example compresses capability and bias into one number c. A type with capability c can execute any action at or below c, so its action set is A(c) = (-\infty, c]. Every type sees the state perfectly. The verification order is therefore just the usual order on capability: a higher type can certify a lower one, but a lower type cannot certify a higher one.¹

Now give a report \hat{c} a permission cap \bar{a}(\hat{c}). The permitted actions are (-\infty, \bar{a}(\hat{c})], which can stand for a spending limit, a tool-use limit, or an authorization boundary. A type c can report any \hat{c} \le c. If a lower report buys a looser cap, the higher type has a profitable reason to sandbag. Truthful reporting therefore requires the cap schedule to be weakly increasing with the reported capability.¹

That condition decides whether evaluation is useful. If more capable types are less biased, the human’s ideal cap rises with capability, so the evaluated cap can match the complete-information outcome. If bias rises with capability, the human wants tighter caps for the types most able to imitate lower ones. The optimal result then pools control into a common cap, and evaluation has the same payoff as that common cap. For a nonmonotone bias profile, the paper irons squared bias into a decreasing schedule before recovering the cap.¹

The best cap sees the square of the bias

The second application removes capability uncertainty. The AI can take any action, but its bias b is unknown. The designer’s belief has mean \bar{b} and variance \sigma^2. In this example, “alignment” means lower mean bias. “Interpretability” means lower dispersion in the designer’s belief about that bias. It is a statement about uncertainty in the model, not a claim that the paper has measured interpretability inside a neural network.¹

The square matters. A type with a modest average bias but a wide range of possible biases can still pull the cap down. The human wants the AI to track the state, but a high-bias realization makes unrestricted action costly. The cap protects against that realization while preserving some discretion for a low-bias realization. Under the paper’s maintained condition, the tradeoff is summarized by average squared bias rather than mean bias alone.¹

Figure 8 makes the control choice move. The authors set a synthetic frontier where a technology parameter \kappa lowers mean bias while increasing residual uncertainty. Under full discretion the preferred point is about \kappa = 0.53. With the optimal cap it is about \kappa = 0.82. The cap first loosens as mean bias falls, then tightens as uncertainty grows. These are values on the paper’s illustrative frontier, not measurements of models or training runs.¹

This gives alignment and interpretability a precise interaction. They are substitutes in the instrument because the cap sees only the sum \bar{b}^{2} + \sigma^2. They are complements in value because a better-aligned AI makes extra discretion safer, while a better understood AI makes discretion more useful. The joint design claim is about the policy selected after the technology is known, not a claim that interpretability mechanically creates alignment.¹

The control result depends on the menu

The multi-agent examples apply the same logic to oversight. In the weak-to-strong model, a weak monitor observes the strong actor’s bias but not the state. The human chooses whether the monitor may approve or reject the actor, delegate a permission set, or choose a state-independent reward schedule. More expressive instruments let the monitor shape the strong actor’s incentives more precisely. The paper shows, within its model, how reward design can reach the monitor’s first-best action, and constructs a reward menu that robustly induces the human’s first-best action when the monitor’s bias is bounded.¹

The assumptions do the work. The monitor sees the strong actor’s bias. Rewards can be chosen from a very rich class. Peer discipline assumes co-player beliefs are informative and rewards are free and unbounded. Coupled rewards approach first best when biases lie on a known low-dimensional curve. The universal type space supplies arbitrarily rich higher-order beliefs as a benchmark. These are useful constructions, but they are not measurements of whether a weak model can diagnose a strong model or whether either model will optimize the prescribed objective.¹

AI Control asks the neighboring empirical question. Greenblatt and colleagues replace an untrusted model with a red-team attack policy and test whether a safety protocol remains safe despite intentional subversion. In their code-backdooring testbed, one undetected backdoor is a safety failure across the sequence of tasks.⁹ Christiano, Shlegeris, and Amodei’s Iterated Amplification work starts from a different bottleneck: a human cannot directly evaluate some hard tasks, so a training signal is built by decomposing them and coordinating copies of a learner.¹⁰ The focal paper contributes a language for the incentives in both settings. It does not supply their missing empirical validation.