The research team at JetBrains (JetBrains Research) has revealed a fundamental gap in how AI-powered coding agents are evaluated, asserting that current benchmarks do not accurately reflect the way developers interact with these tools in real-world work environments. Through analyzing 4,782 developer work sessions using integrated development environments (IDEs), the researchers found that current models are tested under ideal conditions that do not resemble the complex reality of daily programming.
The Gap Between Evaluation Benchmarks and Practical Reality
The study showed that popular benchmarks such as SWE-Bench Pro rely almost entirely (93.8%) on bug-fixing tasks or adding new features presented in a single, fully detailed prompt. However, analysis of real production sessions revealed that less than half of what developers request from coding agents actually pertains to fixing bugs or adding features.
The remainder of developer requests in reality are distributed across diverse tasks including:
- Inquiring about code architecture and understanding how specific parts of a project work.
- Planning next steps in the development process.
- Code review and refactoring.
- Executing code commands and troubleshooting.
Additionally, the research found that half of real sessions consist of a single message, while the remaining sessions extend to include multiple interactions, with requirements arriving at the agent gradually and in fragments. This is lacking in the traditional SWE-Bench benchmark, which presents the complete task in a single polished message.
SWE-TaskFlow Framework: Toward a More Accurate Simulation of User Behavior
To address this shortcoming, JetBrains introduced a new framework called SWE-TaskFlow. This framework does not aim to replace current benchmarks, but rather to transform them to become more realistic through three main mechanisms:
- Task Splitting: A single programming task is split into three consecutive requests, such that the agent does not see the second request until the first is completed, testing its ability to handle gradual requirements.
- Repository Questions: Questions are generated that require the agent to explain parts of the existing code before or during task execution, which is a very common behavior among real developers.
- Task Flow Alignment: Utilizing algorithms to optimize the sequence and types of messages to match the distribution of real sessions based on the TFAS metric created by the team.
Experimental Results: Cost Versus Accuracy
The researchers tested the new framework on several advanced models, including GPT-5.6 Luna, GPT-5.6 Terra, and Gemini 3.7 Flash. The results showed that splitting tasks into multiple messages led to an increase in operational cost by an average of 2.0 to 2.6 times per task. This increase reflects the additional effort exerted by the agent when requirements arrive in a fragmented manner.
Technically, the models' success rates in resolving tasks (Resolve Rates) were not significantly affected when tasks were split, dropping by a slight margin ranging between 4 to 5 percentage points. This indicates that the greater challenge lies in increased resource and time consumption rather than the difficulty of the task itself.
What Changes Practically for Developers and Companies?
The importance of this research lies in shifting the focus of the AI industry from merely "solving programming problems" in a single step to building "interactive assistants" capable of managing long dialogues and understanding changing context. For companies developing models, this means the necessity of improving agents' long-term memory capabilities and their ability to process sequentially arriving information without losing the ultimate goal of the task.
SWE-TaskFlow also opens the door to the possibility of stage-by-stage verification of the agent's work quality; instead of evaluating only the final outcome, developers can now evaluate the agent's performance after each message, helping to detect deviations from the correct path early in the AI-assisted programming process.