WAKMan: A Cache Manager for LLM Servers That Picks Its Own Strategy and Boosts Hit Rates by 5 Points

Why one rule doesn't fit all

Large language model servers spend a significant share of their compute reprocessing the same context over and over. When two requests start with an identical chunk of text — a system prompt, say, or shared conversation history — the server can compute the model's internal "keys" and "values" for that chunk once and then simply pull the ready result from GPU memory instead of recomputing it. This is called prefix reuse, and it meaningfully speeds up responses while cutting serving costs.

The catch is that GPU memory is limited, so space for this cache has to be freed up constantly by evicting older entries. The rule used to decide what gets evicted directly determines how often the server can actually reuse previously computed data. Researchers from the University of Maryland and Capital One have shown that no universal rule exists — and they've proposed a system called WAKMan that picks the right strategy on its own, by watching real traffic.

Traffic to LLM servers is typically mixed: one-off questions with no memory of previous messages, multi-turn conversations with chat history, and agentic scenarios where an AI system executes a rapid series of steps on its own.

These three request types have fundamentally different reuse patterns:

• Agentic steps return to shared context almost instantly, within seconds, which favors the "evict the oldest" rule (LRU) — the current default in the popular vLLM engine.
• Popular fixed-prompt requests (like system prompts for frequently used services) and long conversations return to their context after many unrelated requests in between. For them, frequency matters more than recency — which is where S3FIFO wins, since it shields frequently requested blocks from being pushed out by one-off "noise."

The authors tested both rules on 20 artificially mixed request streams with varying shares of one-off, multi-turn, and agentic scenarios. LRU beat S3FIFO on agent-heavy mixes, delivering up to 12% more cache hits, but lost by up to 22% on mixes dominated by conversations and repeated queries. What's more, even with a fixed traffic composition, changing just one parameter — say, the average number of turns in a conversation, or the length of a one-off request's input text — was enough to flip the winner. On the exact same request mix, LRU won with short, 2-turn conversations, but with 8-turn conversations S3FIFO took the lead by a 15.4-percentage-point margin.

How WAKMan works

The idea behind WAKMan is to stop guessing the traffic type in advance and instead keep testing it in practice. The system runs two "shadow" copies of the cache alongside the real server — one following the LRU rule, the other S3FIFO — and replays the log of recent requests from the last time interval through both. Whichever strategy produces more matches against already-cached data on that same request set wins, and the live server switches to it until the next check.

To do this, WAKMan doesn't need to know anything about request content, type, or turn number in a conversation — it relies solely on the data-block hashes that vLLM already computes during normal operation. That matters because, in practice, a server often has no way of determining a request's type at all: conversation history gets stitched together client-side, and privacy and data-separation requirements prevent the server from "reading" request content to guess its nature.

Results and limitations

In experiments with a Gemma model deployed via vLLM across four NVIDIA RTX PRO 6000 GPUs, WAKMan delivered an average of 5 percentage points more cache hits than vLLM's standard LRU rule. Among the three strategies tested — pure LRU, pure S3FIFO, and dynamic WAKMan — WAKMan showed both the smallest average and smallest maximum gap from an ideal, infinitely large cache that could theoretically store everything.

The approach's main weakness is response lag: the decision to switch strategies isn't made instantly, but at the end of a completed interval, so if traffic patterns shift sharply right after a switch, some requests in that interval still get served by a suboptimal strategy. The authors also note that other cache eviction strategies can be added to the system, as long as there's a way to simulate their behavior without real hardware — meaning the pool of "candidate" strategies isn't limited to just LRU and S3FIFO.

For teams running LLM products with mixed traffic — where some users carry on long conversations, some run agentic workflows, and others just ask one-off questions — the practical takeaway is clear: a hard-coded caching strategy is almost guaranteed to lose performance on some slice of traffic, and it's worth building adaptive switching into the infrastructure itself rather than picking "the one correct" rule once and for all.