How much KV cache can you prune before quality breaks?
During autoregressive generation, every new token adds keys and values to a cache so the model does not recompute the entire history. That makes generation practical, but the cache grows with context length and attention must read more history at every step. Not every old token may be equally useful. If some can be discarded, memory use and attention time can stop growing. The question was: how much KV cache can we evict before model quality degrades, and what memory and speed do we recover?
Why this matters
Long contexts can exhaust GPU memory even when the model weights fit easily. They also slow every subsequent decode step. A useful eviction policy therefore has to balance three outcomes at once: prediction quality, live KV memory, and decode speed.
Main finding
Keeping 4,096 of 6,144 tokens with four attention sinks changed WikiText-2 perplexity from 11.213 to 11.219, only 0.06%, while saving one-third of live KV memory. At a 120,000-token original context, a 512-token window held KV memory at 14 MiB and decoded 6.84x faster than the full cache.
The recent-window policy preserved quality better than the simple attention-score policy at every equal budget in this workload.
What we did
I implemented two eviction policies on HuggingFace’s DynamicCache:
- attention sinks plus a recent sliding window
- four attention sinks, 32 pinned recent tokens, and older tokens selected by cumulative attention score
Quality was measured on a fixed WikiText-2 token stream. The first 4,096 predictions warmed the cache, and perplexity was computed over the following 2,048 predictions. Cache budgets were 4,096, 2,048, 1,024, and 512 tokens, with the full 6,144-token context as the control.
For systems behavior, I measured exact starting contexts from 512 to 120,000 tokens using correctly shaped synthetic KV tensors. This isolated the real one-token attention cost without replaying an impractically expensive long prefill for every configuration. Timings included the policy’s cache-compaction cost.
Detailed results
| retained tokens | recent-window perplexity | memory saved |
|---|---|---|
| 6,144, full cache | 11.213 | 0% |
| 4,096 | 11.219 | 33.3% |
| 2,048 | 11.421 | 66.7% |
| 1,024 | 11.724 | 83.3% |
| 512 | 12.597 | 91.7% |
The first clear quality trade-off appeared at a 2,048-token budget. It saved two-thirds of the KV memory for a 1.86% perplexity increase. Below that point, the quality curve became noticeably steeper.
The attention-score heuristic performed worse than the recent window at every budget. On this contiguous language-modeling stream, attention received in the past was not a better predictor of future usefulness than simple recency.
Full-cache decoding slowed as context grew. Every bounded cache remained near 28 to 29 tokens per second once it reached its budget.
At 120,000 tokens, the full cache occupied 3,281.6 MiB and decoded at 4.26 tokens per second. Windows of 512, 2,048, and 4,096 tokens stayed near 28 to 29 tokens per second while using 14, 56, and 112 MiB of live KV memory.
What we learned
KV-cache pruning has a useful middle region. A model can discard a meaningful fraction of history before quality changes appreciably, but aggressive pruning eventually produces a steep penalty. The best token-selection rule is also workload-dependent; a plausible attention-based heuristic did not beat a simple recent window here.
Bounded caches solve two systems problems simultaneously. They cap memory and keep attention work from increasing indefinitely with the original context.
Limitations
The quality result comes from one WikiText-2 slice, one model, and one GPU. The
attention-score policy is a simple heuristic rather than an optimized published
method. Long-context speed used synthetic KV values, which are valid for runtime
and memory but not for model quality. Cache compaction used index_select; a
production ring, paged, or static cache could reduce its copying overhead.
Future work
The next experiments should compare stronger heavy-hitter and learned policies, evaluate retrieval and question-answering tasks, and integrate pruning into a paged serving engine with a cache layout that avoids repeated tensor copies.
Technical evidence
Read the complete report, code, raw measurements, and reproduction instructions on GitHub.