🔍 Read the full analysis: FrontierCode: When An AI Opens A Pull Request, Who Decides What’s Next? on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks graded for whether maintainers would accept the resulting change. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts, showing that human review remains necessary. The results also show wide differences in task costs and a close showing from several open-weight models.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark that grades whether a code change would be accepted in a real repository rather than simply whether tests pass. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in just 57.7% of attempts, indicating that even the leading system still needs human review before its pull requests are merged.
The results were added to Artificial Analysis’s evaluations on October 12, 2026. FrontierCode’s 100-task Main set is drawn from a broader 150-task benchmark built around issues in established software repositories. Artificial Analysis ran each model through the same open-source agent harness, Stirrup, allowing up to 250 turns per attempt and three attempts per task.
Claude Opus 5.5 (max with fallback) scored 52.2%, ahead of GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The separate pass-rate measure shows how often a model cleared all blocking checks: Opus did so in 57.7% of attempts, GPT-6.1 Sol in 54.7%, and Fable in 54.3%. Those figures are not the same as the overall benchmark score.
FrontierCode checks patches across hidden tests, command checks, tests written by the agent, adaptive tests, limits on files and diff size, and an LLM code review against maintainers’ criteria. Failing any blocking check scores zero for that attempt. Artificial Analysis used GPT-6 Luna (medium) for the review step; the benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up solutions.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results address a practical question for software teams considering AI-generated pull requests: who decides whether a patch is ready to merge? FrontierCode’s scoring reflects concerns that a patch can pass tests yet still be too broad, poorly tested, or inconsistent with maintainers’ expectations. Its results suggest that current systems do not reliably meet those broader acceptance criteria without oversight.
The leaderboard also shows why model choice may depend on the team’s priorities and budget. Artificial Analysis puts MiMo-V2.6-Pro at 45.4%, less than half a percentage point below Claude Sonnet 5.5 at 45.8%, while DeepSeek V4.1 Flash scored 44.0% and Kimi K3 scored 43.6%. These results place several open-weight models among the higher-scoring systems, though they do not establish how any model will perform on a particular company’s codebase.
Costs vary sharply in the reported results. GPT-6 Luna (max) scored 41.9% at $0.09 per task, while GPT-6.1 Sol scored 49.1% at $0.81 and Claude Opus 5.5 scored 52.2% at $10.15. The comparison makes the trade-off visible: the top score came with a much higher reported cost, while a cheaper model may suit teams that value cost over the final points on this benchmark.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules intended to prevent systems from finding solutions online. According to the source report, more than 20 maintainers wrote tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost; each task took more than 40 hours to create. The task set is private so models cannot have encountered those exact tasks in training.
Unlike a benchmark that reports only whether tests pass, FrontierCode combines blocking checks with quality checks. The review evaluates the patch against maintainers’ criteria, while scope rules penalize changes outside relevant files or diffs that are excessively large. An attempt that passes all blocking checks receives a score from 70% to 100%, with quality determining the result within that range. Nine of the 100 tasks were not solved by any model on any attempt, according to the source material.
Artificial Analysis’s results also diverge from rankings on its Intelligence Index. In the comparison provided, Claude Sonnet 5.5 ranks second among the listed models on that index but fifth on FrontierCode; Grok 4.7 falls to 18th on FrontierCode. DeepSeek V4.1 Flash and GPT-6 Luna rank more strongly on the coding benchmark than their general intelligence scores might suggest. The figures show that general-model rankings do not directly predict repository-patch performance.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Limits of the Current Results
The results do not show how often teams would accept AI-generated changes in their own repositories. FrontierCode’s private task set covers a defined group of projects, and Artificial Analysis tested the models using one shared harness, turn limit and review model. Performance may differ with different repositories, agent setups, review procedures or task types.
Several models had no reported FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every relevant system. The source also does not specify how scores or costs would change across repeated benchmark runs beyond the three attempts per task.
The benchmark’s LLM review is itself part of the scoring process, and the source does not provide enough detail here to assess how closely that review matches decisions by human maintainers. The results therefore quantify performance under FrontierCode’s particular criteria; they do not prove that a model’s patch is safe or suitable for production.
Awaiting More Model Results
The immediate next development is whether models without published results are added to the leaderboard. Artificial Analysis had not reported results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview as of October 13, 2026. The source report singled out Muse Wonder Full as a system whose eventual FrontierCode result would be of interest, but did not give a release date for further evaluations.
For engineering teams, the published figures offer a benchmark comparison, not a decision to hand over merging authority. Teams considering AI-generated changes can use the results to compare score, blocking-check pass rate and cost per task, then assess performance against their own review rules and repositories. No change to FrontierCode’s methodology or next evaluation date was announced in the source material.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes using tests and other checks, including patch scope, regressions, the quality of the agent’s tests and an LLM review against maintainer criteria. Its central aim is to judge whether a change meets repository-style acceptance standards, not just whether tests pass.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts, a distinct measure from its overall score.
Does the benchmark show that AI pull requests can be merged without review?
No. The results show that the leading model cleared all blocking checks in 57.7% of attempts under the benchmark setup. They do not establish that AI-generated changes are ready to merge without human review.
Which open-weight models scored well?
MiMo-V2.6-Pro scored 45.4%, DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6% in the reported results. MiMo-V2.6-Pro was the highest-scoring open-weight model identified in the source report.
What remains unknown?
Results were not available for several models, including Gemini 4 Argon and Muse Wonder Full, as of October 13, 2026. It is also unclear how closely benchmark results would match performance on a specific team’s codebase and human review process.
Source: ThorstenMeyerAI.com
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
