Research published by NVIDIA concluded that the software layer surrounding an AI model may affect an agent’s performance more than the underlying model itself, particularly when the task is extended and requires a long chain of decisions. This layer includes tools, memory management, context-use rules, feedback mechanisms, as well as the runtime environment, skills, and libraries available to the model.
The experiment showed that using a custom harness designed to handle memory and incorporating a supervisory component enabled Claude Opus 5 to achieve a perfect score of 100% on the ARC-AGI-3 interactive reasoning test. The test includes two-dimensional games without direct instructions, and the model must infer how to play and win, in a task resembling a human’s attempt to understand a new game. Before using the harness, the model scored 30%, the highest result among the models tested without it.
What Does the Harness Add to the Model?
The language model plays the role of the “brain” that makes decisions, but it does not operate alone within the agentic system. The harness determines how information is stored and retrieved, which tools the model can use, and how steps and results are organized and reviewed. According to Adel El Hallack, Vice President of Products in NVIDIA’s AI unit, an agent is not limited to an API for the model, but consists of the model, the scaffolding surrounding it, the tools, the runtime environment, the associated skills, and the libraries.
These elements are especially important in long-horizon tasks, which require linking numerous decisions over hours or days to reach a complete result, rather than providing a single answer to a short request. The longer the path, the greater the likelihood of losing context, repeating steps, or veering toward an unproductive path.
The Supervisor’s Role in Preventing Drift
The most prominent feature of NVIDIA’s design was the addition of a supervisory agent alongside the primary agent carrying out the task. According to El Hallack’s description, this component operates somewhat like an executive manager: it pushes the agent in the right direction when it gets stuck, alerts it when it begins a path that could lead to a dead end, or asks it to re-examine a path it has previously tested.
The idea of a supervisory agent is not new, but many current agent tools rely on only a single layer in the harness, such as Claude Code, Codex, and Hermes. The article says that NVIDIA researchers created an expanded harness for this test called Agentic Variation Operators (AVO), enabling them to test memory management and supervision in a more advanced manner.
What Does This Mean for Agent Users?
The results show that choosing the strongest or newest model alone is not enough to build a reliable agent. The same model may achieve radically different results when run inside different harnesses, because the design of memory, context management, and decision review changes how it operates in practice. This matters to teams developing agents for programming, document editing, or carrying out multistep procedures, since engineering the system surrounding the model may be a decisive factor in accuracy and stability.
This result follows other indications of the difficulty of long-horizon tasks. In April, Microsoft tested 19 large language models on document-editing tasks and found that all the models, including advanced ones, introduced errors into the files. There have also been cases in which models making sequential decisions deleted users’ files or entire databases, or resorted to criminal behavior such as collusion and hacking to achieve their goals.
The ARC-AGI-3 test carries additional significance because OpenAI models recorded results below 10% on it, prompting the company to conduct its own research the previous month. OpenAI found that modifying just two settings in the harness tripled the results of its models, but they did not reach the 100% result achieved by NVIDIA’s design in the aforementioned test.
Impact on Cost and Openness
NVIDIA does not present this result as a new product. The company offers, under the Nemo brand, open and commercial components that can be used to build harnesses for agents, but the AVO research itself is not an announcement of a new product. The findings are consistent with research published by Databricks in July, which indicated that the harness can have a significant effect on the cost of operating AI. According to a statement by Ali Ghodsi, Databricks’ CEO, using an unsuitable harness may double the cost even when running the same model.
The broader message in NVIDIA’s research is that open harnesses give users greater ability to tune the system, just as open models give them more room for control. The company believes that having control over the harness, infrastructure, and runtime environment is necessary to advance agentic systems more safely. However, the findings do not eliminate the importance of the model; rather, they indicate that an agent’s final performance is the product of the model and the layer that organizes its memory, tools, and decisions—not the result of the model alone.