Researchers release MERIT, a cost-accounted benchmark showing long-term memory helps tool-using LLM agents but retrieval-based updates are unreliable

Researchers released MERIT, a new benchmark and open evaluation harness testing whether long-term memory actually improves task outcomes for tool-using LLM agents, rather than just recall on chat transcripts. Across 23,440 scored episodes (about $42.57 in compute) on GPT-4.1, GPT-4.1-mini and Claude Haiku 4.5, with a spot-check on Claude Sonnet 5, task success rose from a verified 0.00 floor to between 0.55 and 1.00 depending on memory design. Embedding-based retrieval memory was unreliable on updated facts (0.30-0.95 success, and correctly retrieved facts were only acted on 55% of the time), while structured fact stores and LLM-summarization memory scored 0.70-1.00. Full conversation replay was never the cheapest option tested.

This gives developers and enterprises building agent systems a concrete, cost-accounted reason to avoid naive embedding-retrieval memory for facts that change over time, and to prefer update-on-write approaches instead. The 55% figure is the standout: it means even when retrieval works correctly, agents frequently fail to act on the retrieved information, a distinct and under-discussed failure mode from retrieval accuracy itself. Impact is real but conditional: it depends on practitioners adopting this benchmark or its methodology rather than existing ones like LoCoMo, and on the finding holding up outside the specific tool-use tasks tested.

The claim of being "the first" to measure marginal utility of memory under cost accounting is a positioning statement from the authors, not an independently verified fact, and should be treated cautiously. This is a single arXiv preprint with no peer review or independent replication, tested on only three-plus models on synthetic tasks, so the "up to 60-point swings" framing is accurate for this test setup but should not be read as a universal law of agent memory.

1. 01Whether other labs or independent researchers replicate the MERIT results or cite the benchmark in follow-up work within the next 3-6 months
2. 02Whether any production agent framework (LangChain, LlamaIndex, AutoGPT-style tools, or vendor agent SDKs) publicly changes its default memory architecture citing this or similar findings
3. 03Whether the 55% correct-retrieval-but-not-acted-upon failure mode is confirmed or contested by other evaluations of tool-using agents