Swapping the model moves AstaBench’s overall score twice as far on average as swapping the loop

What a score of 58 averages

AstaBench, a benchmark from the Allen Institute for AI (Ai2), gives AI agents a scientist’s work: find and read papers, run other people’s code, test hypotheses in data, and carry a research idea through to a report. Its test set holds 1,918 problems in 11 benchmarks and four categories. A score runs from 0 to 100 and averages the four categories equally. Benchmarks grade by different metrics, six of them with a language-model judge, so a 58 does not mean 58% of problems solved.¹

Between two general-purpose agent loops, the model explains 89% of the variance in overall score across seven models, against 2% for the loop and 9% for their interaction. A specialized agent takes most of its lead from one category, End-to-End Discovery. In ReAct’s 2025 runs, token price does not predict cost per problem, because tokens used vary by more.²

Fig. 1 splits the top score on Ai2’s leaderboard, ReAct on Claude Opus 4.7.³ ReAct is Ai2’s baseline loop: call the model, run its tool calls, repeat.¹

Between general loops, the model moves the overall score further

Smolagents Coder, the second general loop, acts through Python blocks in a sandbox, where ReAct uses native tool calls.¹ Across a row of Fig. 2, a loop swap moves the overall score 5.3 points on average. Between two rows under one loop, a model swap moves it 13.0 or 11.3.

ReAct leads on six of the seven, and five gaps clear their intervals, one of them Gemini 2.5 Flash’s reversal. The seven span releases from August 2024 to August 2025 and tiers from flagship to small, so part of the model effect is a tier effect. Between the top two, GPT-5 and Claude Sonnet 4, a model swap moves 5.9±3.8 and 0.4±3.3 under the two loops, the size of the loop swaps on those models. On Code alone, a mean loop swap (13.2) nears a mean model swap (17.8 or 14.1).²

Asta v0’s lead, mostly from End-to-End Discovery

Asta v0, Ai2’s specialized agent, routes tasks to sub-agents such as Asta Panda for end-to-end work.¹ It mixes five models, so Fig. 3’s last row holds the model fixed.

Literature’s +7.9 also clears its interval, and Code’s −7.4 offsets it. An End-to-End score is the share of rubric items a judge finds met. Full-task completion in Ai2’s AstaBench paper ran 0 to 5%.¹ Princeton’s Holistic Agent Leaderboard sets task-specific designs against a generalist. There the model-first ordering flips on SWE-bench Verified Mini (28.2 points for a design swap, 19.7 for a model swap) and holds on CORE-Bench Hard (8.3 and 15.5).⁵⁶

Cheap tokens at both ends of the cost range

Effective price is reported cost over total tokens, cache discounts included. Llama 4 Scout under ReAct uses 3.69 million tokens at $0.184 per million: 3.69 × 0.184 = $0.68. Against o3 it pays a 5.6th of the price, uses 23.4 times the tokens and costs 4.15 times as much.

The AstaBench paper says Llama 4 Scout and Gemini 2.5 Flash “often take more steps or get stuck in loops,” which result files, holding no step counts, cannot test.¹ GPT-5 Mini and Claude 3.5 Haiku have cheap tokens and the two lowest costs. Across the nine 2025 ReAct rows, rank correlation with cost is −0.13 for price and +0.88 for tokens. Across Smolagents Coder’s eight, they reverse, at +0.57 and +0.02.²

Only three designs have an overall cost

They are ReAct, Smolagents Coder and Asta v0. All 24 costed rows are Ai2’s, none newer than April.³ The newest uploads, on 27 October, come from outside: Andrew Borthwick’s RoboPhD evolves multi-model agents by machine to hit a stated price.⁷

RoboPhD matches Asta Paper Finder on score and cost. It mixes models, so this isolates no design effect, and with 16 public submissions, selection cannot be excluded. Each upload covers one benchmark, so Fig. 4 has no design from outside Ai2.²