FrontierMemory measures what agents really remember

Letta publishes FrontierMemory, a benchmark evaluating whether an agent learns from experiences, not just recalls. Clear gaps exist between harnesses, models, and costs.

Letta published FrontierMemory on Wednesday, a benchmark testing whether AI agents learn from past experiences, not just recall stored information at the right time. The result matters for developers, as an agent’s memory depends as much on the model as on the harness, the software layer orchestrating its actions. In initial tests, gaps are clear, with Letta’s Dream at 73.3% and OpenAI Agents SDK at 40% on the same model. In short, agents’ memory is not yet as consistent as a good intern who takes notes.

Le benchmark montre que le harness pèse presque autant que le modèle

FrontierMemory starts from a simple idea: knowing something is not enough; it must be used at the right time on a subsequent task. Letta reconfigured coding and knowledge work tests to force agents to build on previous attempts, with only 15 tasks in this first version. According to Charles Packer, co-founder and CEO of Letta, classic benchmarks are already saturated because they mostly measure fact recall, whereas the real issue is continual learning in the context of an agent.

Scores vary greatly depending on the execution infrastructure. Letta Dream reaches 73.3%, Prime Agent 60%, OpenAI Agents SDK 40%, Claude Managed Agents 33.3%, and Hermes Agent 26.7%. When Letta compares Dream and OpenAI Agents SDK with the same GPT-6.1 Sol model, Dream scores 60% versus 40%, suggesting that the execution framework significantly changes performance. The message is clear: the engine matters, but so does the gearbox.

Des modèles plus chers peuvent mieux fabriquer la mémoire

The benchmark does not only measure final performance. It also evaluates models’ ability to generate useful memories for future tasks. On this point, Claude Opus 5.5 and GLM-5.3 tie for first place with 73.3%, ahead of Kimi K3 at 66.7%, then GPT-6.1 Sol and Gemini 4 Argon at 60%. Muse Wonder drops to 46.7% and Grok 4.7 to 33.3%.

But the ranking does not tell the whole story. OpenAI retains a cost advantage in Letta’s published figures: GPT-6.1 Sol is estimated at $0.77 per task, versus $2.80 for Claude Opus 5.5 and $3.58 for GLM-5.3. In other words, a more expensive model can produce better memories, while a less costly model can remain more cost-effective in execution. This is the kind of trade-off product teams know well: better, faster, cheaper—choose two [to be verified].

Pourquoi cette mémoire reste imparfaite

Letta also emphasizes the quality of memorized content. In one case involving enterprise pricing data, Opus 5.5 created a useful rule to decide when to consult certain sources, whereas GPT-6.1 Sol generated four distinct skills without the agent retrieving the correct one. In another test, Gemini 4 Argon retained file sizes that were too specific, which helps little when context changes. Conversely, Opus 5.5 produced a more robust instruction, indicating to check both the usual file and emails.

This nuance matters for professional use. Memory that merely archives details can give the illusion of improvement without truly helping the agent act better next time. Letta also warns that an agent can learn bad habits if nothing filters what is retained. The next benchmark version will include longer tasks and continuous memory updates, but at this stage, FrontierMemory already points to a practical limit: the smart agent is not just the one who remembers, but the one who remembers well.

Key points

  • FrontierMemory measures learning, not just recall.
  • Letta Dream scores 73.3% on the benchmark.
  • OpenAI Agents SDK scores 40% with GPT-6.1 Sol.
  • Claude Opus 5.5 and GLM-5.3 lead at 73.3%.
  • The initial test covers only 15 tasks.

By the numbers

  • 73.3% — Letta Dream score, FrontierMemory benchmark, October 2026.
  • 40% — OpenAI Agents SDK score, same test, October 2026.
  • $0.77 — estimated cost per task for GPT-6.1 Sol.
  • $2.80 — estimated cost per task for Claude Opus 5.5.
  • 15 tasks — size of the first benchmark version.

To read