Every time a new model lands, everyone reaches for the benchmark table. I did that for too long and got nothing actionable out of it. A leaderboard tells you which model scored higher on someone else's tasks. It does not tell you which one to hand a tool-heavy refactor, a QA pass on a risky diff, or a research task inside your own codebase.
The assumption I started from: two models don't rank the same on every project. Which one wins depends on the task mix, the conventions, and the context your repo loads into every session. A global score averages all of that away. So the only comparison that helps me is one that runs on my repo, with criteria I define.
#Why this works now
Claude Code sessions can talk to each other. Every terminal running Claude gets a session name, and any session can message another one directly. That turns a few terminals into a small multi-agent setup with zero extra infrastructure.
#Step 1: the criteria
I opened one Claude session and worked out with it what actually defines a model's usefulness on a project: code search precision and speed, edge-case enumeration, recall without tools, decisions under the repo's own rules, QA on a real commit, tool orchestration, MCP usage, research, shipping a small change end to end, token telemetry, and self-correction.
We refined it into one markdown file: 12 dimensions, each with a question, a measurement method, and who scores it. That file is the contract, and it is reusable the day the next model lands.
The criteria file
12 dimensions. Each one has a question, a measurement method, and who scores it. The setter of a task is never scored on it; common tasks are sealed.
| Dim | Question | How it is measured | Scored by |
|---|---|---|---|
| D1Search + speed | Who finds precise code facts faster and correctly? | Single-answer tasks (file:line + value). Time and tool calls taken from the session log, not self-reported. | setter · sealed key |
| D2Edge cases | Who lists more real edge cases for a given function? | Each case peer-verified as real, distinct, and covered or not by existing tests. −1 for an invented case. | peer |
| D3Memory / recall | Who recalls project facts without tools? | Answered from memory, declared, then verified in source. A looked-up answer scores 0. | peer |
| D4Decisions | Who decides better in a realistic scenario? | Scenario → decision, order, refusals, checks. Scored on conformity with the repo's own rules and risks identified. | peer · cites the rule |
| D5QA | Who finds more real defects in a real diff? | Review a commit from history; findings verified in code; offline test plan must be executed. Then a joint mutant run. | peer |
| D6Explanations | Who explains why, not just what? | Qualitative. One sentence per item, no number. | both |
| D7Tool orchestration | Who uses the harness better? | Fan-out task with a verifiable aggregate. Wall time, tool calls, subagents, wasted calls, audited from the log. | peer |
| D8MCP usage | Who uses MCP servers correctly and safely? | Read-only Jira / Confluence tasks with verifiable answers. Error handling counted. Zero writes. | setter |
| D9Research | Who researches external facts better? | Source-verifiable questions. URLs required; peer checks the sources; unsupported claims penalised. | peer |
| D10Task execution | Who ships a small change end-to-end better? | Same spec, own disposable worktree. Tests written and run, diff kept minimal. Peer reviews the diff. No commits. | peer |
| D11Telemetry | Cost and latency of the same work | Tokens and wall time per task window from each session's own log, dedup on message id. Money not estimated. | both |
| D12Self-correction | Who verifies before asserting, and retracts cleanly? | Claims later corrected by self vs by peer; unverified assertions caught. | both |
#Step 2: the setup
Three fresh sessions: one orchestrator, one running Opus 5.5, one running Fable 5.1, both at xhigh effort. The orchestrator got a standing /goal pointing at the criteria file: for each dimension, send the identical prompt to both instances, collect the outputs, score against the criteria.
Isolation was the whole point
- Codebase pinned to a single commit in a read-only git worktree.
- Each model got its own disposable worktree for anything involving edits.
- Each instance wrote results to its own markdown file and never read the other's.
- Shared-task answers were sealed: written to a file, SHA-256 posted before the reveal, verified after.
- MCP read-only, no commits, no pushes, no state changes.
"Tested" means executed, with the command and output pasted. A listed test is not a test.
Telemetry came from each session's own jsonl log, not from self-reporting.
#The output is not a winner
It is a map: on this project, this task shape goes to this model. Different project, different map.
The routing map
On this project, this task shape goes to this model. One round, one code area. Different project, different map.
| Task shape | Lean on | Why |
|---|---|---|
| Fast, precise code-fact lookup | Opus 5.5 | Same accuracy, fewer tool calls, no shell mishaps. |
| Exhaustive edge-case / crash-safety enumeration | Opus 5.5 | Executes every case. Found the fail-open break. |
| QA review where proof matters | Opus 5.5 | Writes and runs probes, refutes its own hypotheses. |
| Spotting the operational consequence a diff hides | Fable 5.1 | Caught the prod default doing DB reads at every full time while writing nothing. |
| Incident decision under a rule set | eitherOpus | Both correct. Opus for evidence preservation: py-spy first, explicit approval gate. |
| Small end-to-end deliverable | OpusFable | Opus for convention conformance and spec-exact diffs. Fable for CLI robustness. |
| Long multi-step build across many files | Fable 5.1 | A third of the turns, heavier batching, about 15% more output tokens. |
| MCP / web research | either | Both exact and read-only. |
And the map does not stay on paper. Claude Code lets you pin a model per agent: a QA-review agent on one model, a long multi-file build agent on another, a fast lookup agent on a third. Each row of the map becomes one agent file, and the routing is enforced by the harness instead of by memory.
#Where the payoff is
You stop spending the most expensive model on tasks a cheaper one handles just as well, and you stop giving a hard task to a model that will burn turns on it.
Every token goes to the model that converts it best on your project. Better output, fewer wasted calls, lower cost, and the whole thing re-runs the day the next model lands.
First published on X · 28 Sep 2026Written by Marchis Rares-Bogdan