On August 30, 2026, JetBrains published an analysis of its methodology for evaluating large language models used inside the Junie coding agent, calling for an approach that goes beyond relying on a single metric: the “resolve rate.” The company believes that knowing whether an agent passed a task’s tests is important, but that alone does not reveal how it reached the result, how much it cost, or whether the code patch was limited and maintainable.
The idea is based on a comparison JetBrains conducted between Claude Opus 4.7 and Gemini 3.5 Flash on its own benchmark. Both models solved the same number of tasks, but Opus required an average of 184 steps and had an operating cost of $2.79, compared with 271 steps and $1.24 for Gemini. The identical final result did not reflect clear differences in execution paths or tool-use efficiency.
What Does the Resolve Rate Conceal?
Junie operates in the context of an issue and a software repository, allowing the model to inspect files, search for symbols, modify code, and run commands and tests. These actions form an “execution path” that can be monitored without claiming that it reveals the model’s internal reasoning. This path can show whether the agent identified the problem’s location before making changes, repeated searches, tested its assumptions, or finished the task without verifying the final patch.
JetBrains therefore proposes a pipeline that combines four evaluation perspectives: functional outcome, execution efficiency, patch quality, and process quality. The specific measurements include test results, runtime, token counts and model and tool calls, cost, the number of modified files and symbols, changes in complexity, repeated file-reading operations, rerunning commands whose results have not changed, and tool-failure loops.
From the Result to the Path Taken to Reach It
JetBrains tested the methodology on four benchmarks comprising 523 tasks when comparing Claude Opus 4.7 and Gemini 3.5 Flash. Opus solved 267 tasks, for a rate of 51.1%, while Gemini solved 254 tasks, for a rate of 48.6%. The result matched on 430 tasks: both models succeeded on 214 tasks and both failed on 216 tasks. The actual difference appeared in only 93 tasks, making behavioral differences more important than the raw gap on the leaderboard.
In an example involving a single task that both models solved, both began by opening a relevant file at step 15, identified the root cause, and conducted comprehensive verification. However, Opus used more targeted searches and began execution after 13 steps, finishing the task in 53 steps with six transitions between exploration, execution, and verification. Gemini, by contrast, inspected the larger unit more broadly, performed its first executable check at step 30, and did not make its first production-code change until step 88. It then required 192 steps and 34 transitions between phases.
Both models passed the tests and modified the same file and symbols modified by the reference patch. However, Opus did not touch any other files, whereas Gemini’s patch extended to four additional files. The evaluation process described it as broad, involving noticeable repetition and some hallucination. JetBrains emphasizes that this example is illustrative rather than an independent statistical result.
Failure Is Not a Single Condition
An analysis of the 216 tasks on which both models failed showed that, according to the evaluation, more than 85% of the cases involved fully or partially identifying the root cause. In one case, both agents understood that text exceeding the symbol limit caused the error, but they proposed truncating the text instead of splitting it into valid parts. In another case, they fixed a download parameter in one path while missing the same problem in an accompanying path.
These cases are practically different from an agent failing to find the responsible component. The cause may be incomplete execution, modifying the wrong layer, failing to comply with the task’s precise contract, or stopping before verification. The execution path can therefore help identify where intervention is needed: improving repository navigation, refining task phrasing, strengthening modification completion, or enforcing a final verification step.
Behavioral Profiles Instead of a Single Ranking
JetBrains concluded that Claude Opus 4.7 was more inclined to identify the underlying cause of ambiguous defects, solving 53 tasks on which Gemini failed. However, 123 of its runs included no executable verification, including 68 tasks considered solved. The company believes that this behavior may leave undiscovered risks involving regressions or edge cases.
Gemini 3.5 Flash, by contrast, was more inclined to run an executable check and use its result to improve the solution when the expected behavior was clear and the responsible component was relatively well defined. However, it showed problems converging on the solution and grounding itself in the repository: it repeated equivalent searches or commands, spent many steps on the build structure, and relied more heavily on undocumented interfaces, dependencies, paths, or test setups. A total of 195 of its runs, or 37.3%, were classified as involving moderate or severe hallucination, compared with 130 for Opus. In addition, 80 of its runs, or 15.3%, showed substantial or severe repetition, compared with 6.5% for Opus.
What Does This Mean in Practice?
In a broader comparison involving GPT-5.5, Claude Opus 4.7, Gemini 3.5 Flash, and Qwen 3.6 27B FP8 across 522 shared tasks, GPT-5.5 achieved the highest resolve rate at 51.5% and was the only model to run an executable check every time. Opus led the patch-quality metrics, while Qwen 3.6 27B FP8 solved 38.9% of the tasks at an operating cost equivalent to 3% of GPT-5.5’s cost.
These figures show that model selection depends on where the costs and risks lie. A model that is stronger at diagnosis may be suitable for an unfamiliar repository or an ambiguous defect, while a model that is more disciplined about verification may be preferable when tests and feedback are available. A low-cost model may become a practical choice when the cost of a failed attempt is limited, even with a lower resolve rate. However, this interpretation does not mean that a model exhibits the same behavior in every environment.
JetBrains cautions against generalizing the behavioral profiles to the models themselves, because the results depended on the specific architecture of Junie. In addition, judgments made by evaluators based on large language models are not “ground truth” and are heavily influenced by a single golden patch, even though correct solutions may use different files or architectural layers. The resolve rate therefore remains a necessary foundation, but its value increases when it is accompanied by evidence about efficiency, modification quality, and workflow, rather than being reduced to a single overall ranking.