SLICE-Mem Cuts AI Agent Memory Token Costs by 92% Without Sacrificing Quality

SLICE-Mem

Cuts AI Agent Memory Token Costs by 92% Without Sacrificing Quality

Researchers from Hong Kong Polytechnic University, South China University of Technology, and affiliated labs have tackled a problem familiar to anyone building long-memory AI agents: to let an agent remember a user across sessions, every message has to be evaluated, parsed, and stored — and that's almost always done using a large language model. The longer the conversation, the more often the model gets called, and the token bill grows faster than the value of what's actually being remembered.

The team's answer is SLICE-Mem, a system that strips the language model out of the most frequent part of the memory pipeline — the online write step — and reserves it only for rare, background cleanup.

How memory selection works

Instead of asking a large model "is this worth remembering?", SLICE-Mem trains a small classifier — a "salience filter." It looks at the user's message along with several preceding turns of dialogue and outputs the probability that the information is worth saving: preferences, facts, plans, promises. It was trained on labels where three different large "teacher" models voted on whether a message mattered, supplemented with existing annotations from open datasets. The selection threshold was tuned experimentally — out of nine candidate thresholds, the authors picked the one that produced the best final result on the Memora Weekly benchmark.

If a message passes the filter, it isn't sent to the model for rewriting. Instead, a standard linguistic parser finds the predicate in the sentence and its related terms: who the actor is, what happened, when, why, where. These pieces are assembled into a compact entry that stays as close as possible to the user's original wording — no model-based paraphrasing involved. If the algorithm can't extract anything useful, it simply stores a cleaned-up version of the original message so the information isn't lost entirely.

What happens to memory later

Working with already-stored entries — finding duplicates, outdated facts (say, someone moved or changed jobs) — happens separately, in the background, and this is where the model finally gets involved. It compares a new entry against similar older ones and decides: keep both as they are, flag one as a duplicate of a more recent entry, mark the old one as outdated (with a pointer to its replacement), or note that the two entries simply complement each other. Crucially, the original text of entries is never rewritten or deleted — only their retrieval status changes.

Benchmark results

On Memora Weekly (an agent memory test that penalizes the use of information that should have been forgotten), running with the Qwen 30B model, SLICE-Mem scored 46.60 versus 46.68 for the comparable system MemOS — a difference within the margin of error. Yet SLICE-Mem spent just 0.536 million tokens versus MemOS's 7.009 million — a 92.4% reduction — and cut memory-building time from 2.378 hours to 0.362 hours, 84.8% faster.

On LongMemEval-S (a test involving questions about multi-session facts, knowledge updates, and temporal reasoning), the accuracy gap was more pronounced: 56.8% for SLICE-Mem versus 59.3% for MemOS. But token savings here were also the largest observed, at 99.5%.

Background memory cleanup has a mixed effect on the final outcome. On Memora Weekly, it pushed accuracy up to 46.74 (with the 30B model) and 47.29 (with the 8B model), but total token consumption rose to 3.34–3.36 million — nearly six times higher than the "pure" online mode without cleanup. So there is a quality gain, but it comes at a meaningful token cost.

What this means in practice

For teams building personal AI assistants with user memory, SLICE-Mem demonstrates a workable trade-off: token consumption can be cut by an order of magnitude with almost no loss in memory quality, simply by replacing the model with a lightweight classifier and syntactic parsing at the most frequent — write — stage. The model is best reserved for rare background cleanup, where it needs to compare entries against each other and resolve contradictions — a task where its contribution yields a statistically meaningful, if modest, accuracy boost.