GitHub believes that measuring the efficiency of coding agents based on the number of tokens in a single invocation can produce a misleading result. A shorter output may force the model to rerun a command or request deleted information, increasing the number of rounds, time, and cost at the level of the complete task. Accordingly, the company reevaluated GitHub Copilot improvements according to the final outcome of the task, not the size of an individual tool response.
In a blog post published by Eric Christensen and Nabales Klesius on September 2, 2026, GitHub explained four changes that it said were developed through offline tests using benchmarks for evaluating agentic coding tasks, then validated through controlled experiments with users before launch. Several Copilot products, including the GitHub Copilot app and code review, use the same underlying infrastructure, while the examples in the blog post came from GitHub Copilot CLI.
Why Isn’t a Shorter Response Enough?
GitHub tested the impact of the RTK, or Rust Token Killer, tool, which shortens shell output before displaying it to the agent. In the test configurations used, deleting some important text led to the original output being reopened or commands being rerun. As a result, the tool’s response size decreased locally, but the task required more tokens and more time on average.
The company emphasizes that this result applies to the integration and workloads it tested, and does not represent a judgment on all RTK configurations or all output-compression methods. The practical lesson is that the evaluation criterion should extend from the user’s request to the final result, while accounting for recovery and rework rounds.
Selective Output Compression
GitHub’s solution was to compress repetitive noise while preserving the information the agent needs. Operational analyses showed that installation, build, test, and lint outputs often contain substantial repetition, whereas code-like outputs and arbitrary command results may contain essential information.
The released version adopted a three-point policy:
- Leave code-like outputs and arbitrary results unchanged, including commands such as cat, git diff, git show, and arbitrary scripts.
- Reorganize search results, such as grep results and file lists, without deleting any result.
- Compress installation, build, test, and progress outputs only when the savings are substantial.
Copilot also retained a direct path for retrieving the complete original output. GitHub continued to monitor use of this path as both a safety mechanism and an indicator that compression had deleted useful information. In the offline tasks where compression was enabled, the company observed no statistically significant decline in task success, while the online experiment slightly reduced average cost without a material decline in the quality metrics being monitored.
Removing Unnecessary Formatting
GitHub achieved one of its clearest savings through the view tool, which displays file contents to the model. The tool added a line number to every line, even though current editing tools rely on matching surrounding code and do not use these numbers in the usual workflow. The company therefore removed these prefixes from file readings, while retaining line numbers where they are useful in diffs and short snippets.
The change reduced model inference cost by about 5% in offline agentic coding-task benchmarks, while success rates remained within the expected variance and editing errors did not increase. In an online experiment with Copilot CLI users, average daily inference cost per user fell by about 3%, with no material decline in the quality or satisfaction metrics measured by GitHub.
Shortening Instructions Without Changing Behavior
The instructions for the task tool had accumulated across tool descriptions, schemas, agent definitions, and system instructions. GitHub used an automated instruction-optimization loop, reducing the text by about half and then testing the behaviors it wanted to preserve.
However, the first online experiment revealed a problem that had not appeared in offline evaluations: guidance on cautious parallelism had turned into a strict scheduling policy, causing independent specialized agents to work sequentially. GitHub stopped the experiment, added a regression test for this behavior, and replaced the allow-and-deny list with a single sentence: “Independent agents can work in parallel; consider side effects.”
The final wording reduced about 1,300 tokens from the task-tool instructions in each round, equivalent to an approximately 1.8% decrease in total instruction tokens per session and a 2.9% decrease in normalized cost per hour of activity, with no decline in quality observed in the measured evaluations.
Eliminating Unnecessary Retrieval Rounds
Agents sometimes perform independent background work, such as running a long shell command in parallel with an investigation conducted by a sub-agent. Previously, completion notifications for this work arrived without the result itself, forcing the agent to make an additional call to retrieve output that Copilot already had. Now, the system collects eligible completion notifications and sends the completed results within the existing tool-result format.
In the example combining a shell command and a sub-agent, completing the work previously required four model calls: two calls to request the results and two calls to process them. After the change, both results arrive together in a single call for processing. This reduced average token-related usage, as measured in AI Credits units, by about 2.3%.
What Changes in Practice?
GitHub’s experiment provides an important rule for developers of coding agents: the safest optimization is not to delete as much text as possible, but to remove work the model does not need in the first place. This includes unused formatting, waiting and retrieval rounds that the system can resolve, and repetition that can be compressed while providing a recovery path.
At the same time, not every result can be generalized beyond the test environment. Narrowing the instructions for file tools, despite succeeding in code review, increased costs in a Copilot CLI experiment, so GitHub did not launch it. Compression of git diff was also removed after benchmarks showed that agents reopened the original output to recover deleted information.
The conclusion established by the article is the need to measure a change at the level of the task, workflow, and product in which it will be used, through offline benchmarks, online experiments, and clear behavioral tests. The effect of these improvements on an individual user, however, remains tied to the type of tasks, tools, and outputs involved, and cannot be inferred from a local decrease in token count alone.