🔍 Read the full analysis: FrontierCode: How Maintainers Decide Whether To Merge AI Code on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis published results for 23 models on FrontierCode, a benchmark that evaluates whether repository changes meet maintainers’ merge criteria, not just whether tests pass. Claude Opus 5.5 ranked first, but cleared every blocking check in 57.7% of attempts; costs and performance varied widely.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark designed to assess whether maintainers would accept an AI-generated code change, and found that the top-ranked model, Claude Opus 5.5, passed all blocking checks in 57.7% of attempts. The results, added to Artificial Analysis’s evaluations on October 12, put a practical limit on claims that coding agents can already produce pull requests ready to merge without review.
Opus 5.5 scored 52.2% on Artificial Analysis’s FrontierCode Main set. GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models from six labs scored between 43% and 52%, while results fell off below that group: Grok 4.7 scored 33.8%, MiniMax-M3 23.0%, and Nemotron 3 Ultra 10.5%.
The benchmark’s score and pass rate describe different outcomes. The score combines blocking checks with additional quality assessments; an attempt that fails any blocking check receives zero. Opus’s 57.7% pass rate means it cleared every blocking check in that share of attempts—not that every patch in those attempts received a perfect score. Artificial Analysis ran the 100-task Main set, drawn from a larger benchmark of 150 tasks.
Each model was evaluated using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. The review component used GPT-6 Luna (medium) as its judge. The source reports that nine of the 100 tasks were not solved by any model on any attempt. The results also show substantial cost differences: Opus averaged $10.15 per task, compared with $0.81 for GPT-6.1 Sol and $0.18 for Xiaomi’s MiMo-V2.6-Pro.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Merge Readiness Remains Uneven
FrontierCode focuses on a decision that matters to maintainers: whether a patch is acceptable in a real repository, rather than whether it can make a test suite turn green. The benchmark checks correctness alongside regressions, the usefulness of tests, scope and diff size, and a review against maintainers’ criteria. That makes its results relevant to teams considering where coding agents can contribute and how much human review their output still needs.
The results do not establish how often AI-written code is accepted in everyday software development; the benchmark is a controlled test, not a measure of production use. They do show that, under these conditions, even the leading model failed at least one blocking check in a substantial share of attempts. For engineering teams, the reported pass rate and cost per task offer a more specific basis for evaluating models than a general capability ranking alone.
Performance also came at sharply different prices. Artificial Analysis identified four models on its cost frontier, where no other model was both cheaper and higher-scoring: GPT-6 Luna (max) at 41.9% for $0.09 per task; MiMo-V2.6-Pro at 45.4% for $0.18; GPT-6.1 Sol at 49.1% for $0.81; and Opus 5.5 at 52.2% for $10.15. The comparison is historical benchmark pricing, not a guarantee of future costs or results.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against looking up solutions online. According to the source material, more than 20 maintainers created tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost. They spent more than 40 hours on each task. The tasks are private, limiting the chance that models encountered the answers during training.
Rather than treating passing tests as sufficient, the benchmark uses six checks: hidden correctness tests; command checks for regressions and cleanliness; reverse tests to confirm an agent’s tests fail on the original code; adaptive tests for complex behavior; limits on touched files and diff size; and an LLM review against maintainers’ criteria. Failing a blocking check means zero, while attempts that pass all blocking checks receive scores from 70% to 100%, depending on quality checks. Cognition’s trajectory scanner also assigns zero if it flags an attempt for looking up a solution.
The published leaderboard is not the full field of available models. As of October 13, the source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result. Artificial Analysis’s comparison with its Intelligence Index also showed that general rankings do not map neatly to this benchmark: Claude Sonnet 5.5 ranked second among the listed models on that index but fifth on FrontierCode, while DeepSeek V4.1 Flash ranked higher on FrontierCode than its general index position suggested.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The leaderboard reflects Artificial Analysis’s run of the 100-task Main set, not all 150 FrontierCode tasks. The supplied material does not provide task-by-task results, score variability across runs, or a breakdown of which blocking checks most often caused failures. It also does not establish how benchmark performance translates to code accepted in live projects, where repository practices and reviewer judgments may differ.
Several prominent models had no result as of October 13, including Gemini 4 Argon, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the table cannot be treated as a complete comparison of current coding models. The supplied report also does not explain whether or when those models will be tested, or whether the benchmark’s task set and scoring rules will change in future evaluations.
As an affiliate, we earn on qualifying purchases.
Further Model Results to Watch
The next clear point of comparison is whether Artificial Analysis adds results for the models missing from the October 13 leaderboard. Additional testing could change the order, particularly among models near the top, where several scores are within a few percentage points. Further releases could also show whether the performance and cost patterns hold across the benchmark’s remaining tasks.
For now, the published figures describe one evaluation setup: the Main set, Stirrup harness, stated turn and attempt limits, and GPT-6 Luna as the review judge. Readers comparing models should treat the scores as results from that setup and check for new benchmark runs before drawing broader conclusions about coding-agent reliability.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It tests whether an AI-generated repository change meets checks that include correctness, regressions, test quality, change scope and an LLM review against maintainer criteria. Passing ordinary tests alone is not enough.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, scoring 52.2%. It cleared all blocking checks in 57.7% of attempts.
Does a 57.7% pass rate mean 57.7% of generated patches were merged?
No. It means the model passed all blocking checks in 57.7% of benchmark attempts. The result does not measure actual pull-request acceptance or merges in live software projects.
Which tested model had the lowest listed cost per task?
Artificial Analysis listed GPT-6 Luna at $0.09 per task, with a FrontierCode score of 41.9%. Those figures refer to the reported evaluation and are not a promise of future pricing or performance.
Were all major models included?
No. The source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
