Evidence
The measurements, and their limits.
Local runs on one model. Each number names its source and its scope. A reduction is a lead until it is repeated.
What was compared.
Atlias's own agent against plain Claude Code 2.1.302, on the same local model (qwen3.5:9b, 32k window), the same tasks, step budgets and grader. Each full screen runs 60 tasks twice: 120 matched pairs. A hidden check decides a solve.
Total tokens include prompts and output. The baseline was reused across screens, so the screens are not nine independent baseline measurements.
What the headline says.
Up to twenty times fewer tokens refers to total-token observations above 20x on some screens. It does not mean every screen is above 20x, and it does not mean no task was lost.
The 5x cached-token figure is a target, not yet measured.
The full record.
Pooled reduction 19.52x: a ratio of pooled token totals across the nine screens, not a mean of the screen ratios. The arithmetic mean of the nine screen ratios is 20.1x.
| Record | Date | Suite and model | Ratio | Atlias or sub-atlias solved | Plain Claude Code solved | Source |
|---|---|---|---|---|---|---|
| Screen 1 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Atlias's own agent against plain Claude Code 2.1.302, same local model, tasks, step budgets and grader. 60 tasks twice, 120 matched pairs. A hidden check decides a solve. Six plain-client runs had capped metering, so its token total is a lower bound. | 22.51x | 86 of 120; tokens 876,810 | 80 of 120; tokens 19,736,971 | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 3) |
| Claude Code with sub-atlias, ten-task smoke, October 16 | 2026-10-16 | 10-task smoke: CanItEdit and HumanEvalFix, one run per arm qwen3.5:9b, 32k window The same Claude Code 2.1.301 driving the same local model in both arms: one plain, one with the sub-atlias trial build and its two Claude Code mods. Ten tasks, one run per arm. A lead, not a screen. The public 4.0.1 sub-atlias package does not ship the two Claude Code mods this trial used. | 2.63x | Claude Code + sub-atlias: 8 of 10; tokens 250,143 | 8 of 10; tokens 659,112 | RESEARCH-FIRST.md section 0f, measured entry dated 2026-10-16 (sprint-lean-r2c, a qualified smoke) |
| Screen 2 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Atlias with reasoning effort set to none for the OpenAI-compatible path. | 22.36x | 83 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 4) |
| Screen 3 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Same configuration family as Screen 2 with the reasoning setting checked against a visible check. | 14.02x | 79 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 5) |
| Screen 4 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. An exact repeat of Screen 3's configuration. | 15.48x | 82 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 6) |
| Screen 5 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Screen 1's configuration rerun from its frozen source. | 19.66x | 89 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 7r) |
| Screen 6 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Screen 1's configuration with a stall stop added. | 19.63x | 85 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 8) |
| Screen 7 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Screen 6's configuration with a shown failure in the next call. | 23.42x | 85 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 10) |
| Screen 8 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Screen 7's configuration with two checker fixes, built default-off and turned on together. | 20.95x | 83 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 11) |
| Screen 9 of 9, October 17 | 2026-10-17 | 60-task screen: CanItEdit and HumanEvalFix, two repeats qwen3.5:9b, 32k window Same comparison and grader as Screen 1. Screen 8's configuration with two more checker fixes. This closes the configuration series. | 22.91x | 82 of 120; tokens not measured | 80 of 120; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-17 (trial 12) |
| Ten-task smoke, October 16 | 2026-10-16 | 10-task smoke: CanItEdit and HumanEvalFix, one run per arm qwen3.5:9b, 32k window Ten tasks, one run per arm. This is a lead, not a screen. Token totals are not kept in this record. Earliest pinned-client comparison on the same model. | 10.61x | 9 of 10; tokens not measured | 8 of 10; tokens not measured | RESEARCH-FIRST.md section 0g, measured entry dated 2026-10-16 (pinned Claude Code 2.1.302) |
On a Claude plan.
Atlias running its own loop on Opus 5.5 on a Claude subscription.1 These runs have no plain Claude Code arm, so they show what a task costs, not a reduction. The two runs differ only in how long the prompt cache lives. Cost is the API-equivalent price the Claude Agent SDK reports; a plan subscriber pays for the plan, not this figure.
| Task | Tests passed | Requests | Prompt tokens | Output tokens | Cost (API-equivalent) | Time |
|---|---|---|---|---|---|---|
| affine-cipher | 16 of 16 | 4 | 26,950 | 5,895 | $0.195 | 71 s |
| beer-song | 8 of 8 | 4 | 41,733 | 3,297 | $0.188 | 43 s |
| book-store | 20 of 20 | not measured | not measured | not measured | not measured | 900 s |
| bottle-song | 7 of 7 | 3 | 21,342 | 3,434 | $0.141 | 45 s |
| bowling | 31 of 31 | 8 | 88,134 | 12,637 | $0.408 | 149 s |
| connect | 10 of 10 | 3 | 18,696 | 3,919 | $0.142 | 46 s |
| Total, 5 measured task(s) | 72 of 72 | 22 | 196,855 | 29,182 | $1.075 | 354 s |
Atlias exec in a fresh folder per task. Before scoring, the original test file is written back, so an edited test cannot score. Tokens are prompt tokens across all requests; cost is the API-equivalent price the Claude Agent SDK reports. book-store passed: the runner wrote the original test file back and ran it after the cap, 20 of 20. The model's own probe script ran to the 10-minute shell timeout, so the run hit the 900-second runner cap before it printed its tokens. Source: Atlias run log polyglot-run-1023/results.jsonl, 2026-10-23
| Task | Tests passed | Requests | Prompt tokens | Output tokens | Cost (API-equivalent) | Time |
|---|---|---|---|---|---|---|
| affine-cipher | 16 of 16 | 5 | 34,904 | 5,773 | $0.172 | 76 s |
| beer-song | 8 of 8 | 4 | 42,387 | 2,619 | $0.134 | 43 s |
| book-store | 20 of 20 | 8 | 70,175 | 12,498 | $0.336 | 229 s |
| bottle-song | 7 of 7 | 8 | 68,475 | 7,346 | $0.228 | 98 s |
| bowling | 31 of 31 | 4 | 34,499 | 6,837 | $0.204 | 76 s |
| connect | 10 of 10 | 2 | 12,413 | 2,523 | $0.089 | 34 s |
| Total, 6 measured task(s) | 92 of 92 | 31 | 262,853 | 37,596 | $1.162 | 556 s |
Atlias exec in a fresh folder per task. Before scoring, the original test file is written back, so an edited test cannot score. Tokens are prompt tokens across all requests; cost is the API-equivalent price the Claude Agent SDK reports. Four tasks were run again after the first attempt reached the plan session limit; the rows are the complete second attempts. Source: Atlias run logs polyglot-run-1023-5m and polyglot-run-1023-5m-b, 2026-10-23
Over the 5 task(s) measured in every run: 1-hour prompt cache $1.075 for 196,855 prompt tokens; 5-minute prompt cache $0.827 for 192,678 prompt tokens.
1 Measured on the developer's own Claude subscription through Anthropic's Claude Agent SDK. Signing in with a Claude plan is not something Atlias offers its users; see the questions on the pricing page. Back to the text