Letta has launched a new benchmark known as FrontierMemory to measure and evaluate the ability of AI agents to continuously learn from past experiences, rather than merely retrieving facts from old conversations. Early tests of the benchmark revealed a wide performance disparity among various agent operation harnesses.
In the results of the benchmark's first release, which was limited to 15 tasks, the Letta Dream framework outperformed others, achieving between 33.3% and 73.3% depending on the models used. It was followed by the Prime Agent framework at 60%, while OpenAI Agents SDK achieved 40%, and Claude Managed Agents and Hermes Agent ranked subsequently with 33.3% and 26.7% respectively.
Why this news matters
The test organizers note that traditional memory benchmarks have become saturated because they only measure the ability to retrieve static facts in current working environments. FrontierMemory bridges the gap between actual recall and using that information to improve performance in subsequent tasks, reflecting a shift toward measuring agents' ability to learn across context without requiring weight updates.
What changes practically?
A comparison of the seven models integrated into the benchmark showed that both Anthropic's Claude Opus 5.5 and GLM-5.3 achieved the highest rate in generating useful memories at 73.3%, outperforming GPT-6.1 Sol, which recorded 60%. However, OpenAI models featured a significantly lower operating cost of about $0.77 per task compared to $2.80 for the Opus model and $3.58 for the GLM model, highlighting a clear trade-off between memory accuracy and operational cost.
Limitations and open questions
The FrontierMemory benchmark is still in an early stage, currently limited to 15 tasks and a single attempt per task, meaning any single result heavily impacts the overall total. The evaluation also focuses on agents learning from past tasks without interactively and directly updating memory during task execution—points the company plans to develop in upcoming versions to include longer tasks and real-time memory reviews.