Why similar AI benchmark scores can hide different models

Similar AI benchmark scores can hide models that work very differently in a real repository. DeepSWE puts Luna Max 2.2 points behind Sol High, but Luna uses almost three times as many agent steps. A pass rate alone cannot tell me how much supervision a model needs or whether I would accept its pull request.

What does a 67% versus 69% benchmark score measure?

It measures per-attempt success in one DeepSWE setup. Sol High’s 69.4% beat Luna Max’s 67.2% by 2.2 points, but that gap is smaller than the run-to-run uncertainty reported in the same data [2]. The result gives me a reason to test both models, not a complete model review.

Measurement	Luna Max	Sol High
Attempt success, %	67.2	69.4
95% run-to-run interval	63.2–71.2%	68.0–70.8%
Attempts scored	448	451
Tasks with ≥1 success, %	90.3 (Best value in this row)	86.7
Average cost per attempt, $	0.61 (Best value in this row)	3.47
Output tokens per attempt, K	73.4	28.5 (Best value in this row)
Agent steps per attempt	101.7	36.9 (Best value in this row)

Datacurve’s DeepSWE v1.1 data records 301 successful attempts out of 448 for Luna Max and 313 out of 451 for Sol High [1] [2]. Those totals produce the headline scores in Figure 1.

DeepSWE controls several important variables. Its 113 tasks come from 91 open-source repositories and cover TypeScript, Go, Python, JavaScript, and Rust [1]. The tasks were written for this evaluation instead of being copied from old GitHub issues. Every model also used mini-swe-agent with the same bash tool and base prompt [5], and a fresh container checked each patch for the required behavior and regressions.

Those controls make the comparison useful, but the result still has uncertainty. Datacurve gives Luna a 95% run-to-run interval of 63.2% to 71.2%, while Sol’s interval is 68.0% to 70.8% [2]. The ranges overlap, so the 2.2-point difference is smaller than the observed variation between runs. Datacurve does not publish a statistical comparison of the two models on the same tasks, so the data does not establish that Sol’s 2.2-point lead is repeatable.

The two configurations also use different effort settings: max for Luna and high for Sol. The ranking changes when I use Datacurve’s empirical Pass@4 measure, which asks whether at least one of the recorded attempts solved each task. On that measure, Luna covers 90.3% of the 113 tasks and Sol covers 86.7% [2]. The per-attempt score favors Sol, while task coverage favors Luna, because the two measurements answer different questions.

Every measurement in Figure 1 belongs to DeepSWE’s tasks and agent setup. My own repository may reward different strengths and expose different failures, which is why the way each model reaches its result matters next.

Why close scores can hide almost three times the agent steps

A pass rate ignores how the model reached the patch, so two similar scores can require very different amounts of developer work. I notice whether a model finds the right files, preserves the existing design, asks before making a risky assumption, recovers after a failed command, and stops when the requested change is complete. DeepSWE does not grade those behaviors separately.

DeepSWE’s raw data does provide one clue. Luna Max averaged 101.7 agent steps and 73,400 output tokens per attempt, compared with 36.9 steps and 28,450 tokens for Sol High [2]. Luna therefore used about 2.8 times as many steps and 2.6 times as many output tokens to reach a similar success rate. The models did not work in the same way.

The displayed cost needs context too. Luna’s trials ran on 7 July 2026, when the recorded token usage cost about $3.03 per attempt [2] [3]. OpenAI cut Luna’s token prices by 80% on 30 July 2026 [4], and DeepSWE recalculated the old usage at the new rate. That is how $3.03 became the displayed $0.61. The lower number is a valid estimate of what the same usage costs now, but it is not what Datacurve paid when it ran the test.

API cost per attempt is also different from the cost of getting a change accepted. The leaderboard price leaves out retries, review time, manual edits, waiting, and any second model used to inspect or repair the first model’s work. A model can cost one-sixth as much per attempt and still cost more in total if an engineer has to supervise every decision.

Developers sometimes call these working differences the “big model smell”: the sense that a stronger model understands the job with less explanation and makes fewer decisions that need to be undone. The phrase is informal, but the experience behind it can be measured. I can count how often I redirect the model, repair its work, or reject a patch that passed the tests.

That turns the impression into concrete questions. Did the model keep track of every constraint, make sensible choices for this repository, and avoid unrelated changes? When it failed, was the problem easy to notice and reverse? A single pass rate cannot answer those questions.

How much can benchmark setup change the score?

In the published examples below, benchmark setup changes moved scores by roughly 5 to 15 percentage points without changing the model [6] [7]. That is larger than Luna and Sol’s 2.2-point gap. A coding benchmark measures the prompt, tools, runtime limits, attempt count, and grader along with the model.

The NeurIPS SWE-agent paper provides a clean example. With the same GPT-4 Turbo model on SWE-bench Lite, a shell-only interface solved 11% of tasks, while the full SWE-agent interface solved 18% [6]. When the researchers replaced repeated search output with a tool that summarized the results, the score rose from 12% to 18%. Limiting the file viewer to 100 lines instead of returning a whole file moved it from 12.7% to 18%. These interface changes added five to seven percentage points without changing the model.

Attempt count changes the headline even more. Six runs of the same GPT-4 and SWE-agent setup averaged 17.94% on the first attempt. Counting a task as solved when any of six attempts passed raised coverage to 32.67% [6]. That 14.73-point increase can be useful if a production system really runs six candidates and can identify the correct one. It is misleading when placed beside a single-attempt result without a clear label.

Infrastructure can add noise too. Anthropic held the Claude model, harness, and Terminal-Bench 2.0 task set constant, then changed CPU, memory, and runtime limits. The uncapped setup scored about six percentage points higher, while infrastructure errors fell from 5.8% to 0.5% [7]. This is why a leaderboard entry should be read as a result for a tested system, not a permanent number attached to a model name.

The four setup changes below all moved scores by more than the headline difference between Luna and Sol.

System change	Before	After	Change
Shell only → SWE-agent interface	11.0%	18.0%	+7.0 pp
Whole-file → 100-line viewer	12.7%	18.0%	+5.3 pp
One → six attempts	17.94%	32.67%	+14.73 pp
Strict → uncapped runtime	Not published	Not published	about +6 pp

This does not make benchmarks useless, but it does make the setup part of the result. Before I compare two scores, I check that they use the same task version, agent setup, resources, attempt count, and grading rule. The Muse Spark 1.3 rescore on the Artificial Analysis index is a recent case of the setup deciding the rank.

Why can a passing patch still fail code review?

A benchmark pass means that a patch passed the benchmark’s checks. That is useful evidence that the code works, but it is not the same as a code review. The checks may miss an incomplete fix, and they may not consider maintainability, fit with the existing architecture, unrelated changes, or the work required before the patch can be merged.

The NeurIPS EvalPlus paper showed how strongly the selected tests can affect the result. The researchers expanded HumanEval’s test suites by roughly 80 times and evaluated 26 models. Under the stronger tests, pass rates for the most affected models fell by 19.3 to 28.9 percentage points, and some model rankings reversed [8]. The answers stayed the same; the larger test suites simply caught more errors.

Repository benchmarks have the same problem. In February 2026, OpenAI audited 138 SWE-bench Verified tasks that o3 failed inconsistently across 64 runs. It found material issues in 59.4% of that selected group, including tests that rejected valid solutions and tests that required behavior not mentioned in the issue [9]. The audit deliberately focused on suspicious tasks, so 59.4% does not describe all 500 tasks. It does show that errors in the tasks and tests can distort differences between highly capable models.

The opposite error matters just as much: a patch can pass the automated tests and still be unfit to merge. METR asked maintainers to review 296 AI-generated pull requests across three repositories and 95 tasks. Their acceptance rate was 24.2 percentage points lower than the automated SWE-bench score [10]. They rejected patches for failures in core functionality, unrelated breakage, poor code quality, and other integration problems. A binary benchmark score hides all of those reasons behind the same “fail” or “pass” label.

more tests in EvalPlus

80×

some rankings reversed

issues in targeted SWE-bench audit

59.4%

not representative of all 500 tasks

automated-to-maintainer gap

24.2 pp

METR review of 296 pull requests

Together, these results show that test quality and human review answer different questions. “The patch passed” is evidence worth keeping, but it does not tell me whether I want the model making changes in my branch every day. Tools can make that review easier to skip, which is why I wrote about Cursor’s Agents Window and how to put the code back in front of you.

What does an average hide? Repeatability and long tasks

An average pass rate combines every success and failure into one number. It does not show whether the model succeeds consistently, whether reliability falls on longer tasks, or whether a failed run produces a small bad diff or damages the working state. Those differences matter when I use the model repeatedly rather than testing it once.

The arXiv tau-bench paper makes the repeatability problem visible with pass^k, which asks whether an agent succeeds on the same task every time across multiple trials. GPT-4o achieved 61.2% on retail tasks in a single-trial view, but its retail pass^8 fell below 25% [11]. A model that works three times and fails the fourth may still have a respectable average. As a daily collaborator, it feels unpredictable.

Task length creates a similar split. METR evaluated agents on 170 tasks with about eight runs per model-task pair. The task length at which a model succeeded 80% of the time was four to six times shorter than its 50% success horizon [12]. A model may therefore earn credit on impressive two-hour tasks at a 50% threshold while remaining dependable only on much shorter work.

This helps explain why hands-on impressions can disagree with a leaderboard. Daily use includes repeated tasks, repository context that is easy to miss, broken commands, ambiguous requirements, review cycles, and the consequences of the worst failed runs. Personal impressions can still be wrong because one polished answer or one disastrous failure may dominate my memory. Instead of choosing between benchmarks and intuition, I need to measure the parts of my workflow that created that impression. Opus 5 is the case I know best: after five weeks of daily use I explained why Opus 5 is not as bad as its online reputation.

How should I compare models for my own work?

I use public benchmarks as a filter. They tell me which models are worth spending time and money on, and a controlled benchmark such as DeepSWE is much better evidence than a launch demo. The final choice still comes from a small evaluation built from my own work.

Anthropic recommends starting an agent evaluation with 20 to 50 tasks drawn from the manual checks, common failures, bug reports, and user requests already seen in development [13]. That is enough to reveal large mismatches without pretending to produce a universal leaderboard. For a close comparison, I would run the same tasks more than once and keep the environment, tools, prompts, effort level, and budget fixed.

Before looking at the outputs, I would define what acceptance means. Tests come first, but they are not the whole rubric. I would also record:

• whether the result was accepted on the first attempt;
• total time to an accepted result, including review and repair;
• active review minutes rather than model latency alone;
• clarifications, redirections, restarts, and manual edits;
• unnecessary files changed and violations of repository conventions;
• severe failures such as security regressions, data loss risk, or unrelated breakage;
• tokens and API cost for the complete accepted result.

For code quality, I would remove model names and review competing patches in random order when practical. I would report results by task type instead of combining bug fixes, reviews, refactors, and long-running feature work into one total. If two models remain close across repeated tasks, I would choose the cheaper one. If its higher review cost consumes the API saving, however, it is not the cheaper model for my workflow.

DeepSWE has done enough to put Luna Max on my test list. My next comparison would keep tasks and tools fixed, then count the full path to an accepted patch: steering, review, repair, and failed runs. If Luna’s lower API price survives those costs, it is cheaper for my work. If it does not, the leaderboard price is not the price I pay. I ran that kind of comparison on my own work in Claude versus Codex on large multi-agent features.