How much KV cache can you prune before quality degrades?
Before reading: If the KV cache is unfamiliar, read Hugging Face’s visual explanation of caching. It explains why an LLM stores keys and values from earlier tokens instead of calculating them again for every new token.
During inference, an LLM first processes the prompt and then enters decode, where it generates one new token at a time. For every earlier token, the attention layers have already calculated key and value vectors. The KV cache stores those vectors so the model can reuse them instead of calculating the complete history again for every new token. Each decoded token adds another set of keys and values to the cache.
This makes the cache grow with the context. Longer caches require more GPU memory, and each later decode step must read more stored vectors. On a smaller GPU, the KV cache can therefore limit the maximum context length even when the model weights themselves fit in memory.
One possible solution is to remove old cache entries that appear less useful. This saves memory and attention work, but removing the wrong tokens can make the model worse at predicting what comes next. The question was: how much KV cache can we remove before quality noticeably degrades, and how much memory and speed do we recover?
Why this matters
Long contexts can exhaust GPU memory even when the model itself fits. They can also make every new token slower. A useful pruning policy must balance three outcomes: prediction quality, live KV memory, and decode speed.
What we did
I tested Qwen2.5-1.5B in FP16 on one RTX 3060 12 GB GPU. I implemented two
policies using Hugging Face’s DynamicCache:
- Sinks plus recent window: keep the first four tokens and use the rest of the budget for the newest tokens. The first few tokens are kept because they can act as attention sinks, meaning that later tokens may continue attending to them even when they are not recent.
- Attention score plus recent tokens: keep the first four tokens, always keep the newest 32, and fill the remaining space with older tokens that had received the most attention. This policy asks whether past attention is a better sign of future usefulness than age alone.
For quality, the first 4,096 WikiText-2 predictions warmed the cache. I then measured perplexity over the following 2,048 predictions (a measure of how uncertain the model is about the correct next token, where lower is better).
For systems behavior, I created correctly shaped KV tensors at starting contexts from 512 to 120,000 tokens. This isolated the real one-token attention cost without repeating a very expensive long prefill. These tensors had the same dimensions and memory layout as a real cache, but their contents were not used to measure quality. The timing includes the cost of selecting the retained entries and copying them into a new cache tensor.
Recent window: one-third less cache, nearly unchanged quality
The 4,096-token budget produced almost the same perplexity as the full cache. The first clear quality cost appeared at 2,048 retained tokens. It saved two thirds of KV memory while increasing perplexity by 1.86%. More aggressive budgets made the quality curve much steeper.
| retained tokens | perplexity | change from full | KV memory | memory saved |
|---|---|---|---|---|
| 6,144, full cache | 11.213 | control | 168 MiB | 0% |
| 4,096 | 11.219 | +0.06% | 112 MiB | 33.3% |
| 2,048 | 11.421 | +1.86% | 56 MiB | 66.7% |
| 1,024 | 11.724 | +4.57% | 28 MiB | 83.3% |
| 512 | 12.597 | +12.35% | 14 MiB | 91.7% |
The recent-window policy preserved quality better than the simple attention-score policy at every equal budget in this workload.
Attention scores did not improve perplexity
The attention-score policy kept older tokens that had received more attention in the past. This sounds reasonable, but past attention did not predict future usefulness better than recency on this continuous text stream.
| retained tokens | recent-window perplexity | attention-score perplexity | better policy |
|---|---|---|---|
| 4,096 | 11.219 | 11.428 | recent window |
| 2,048 | 11.421 | 12.151 | recent window |
| 1,024 | 11.724 | 12.844 | recent window |
| 512 | 12.597 | 13.724 | recent window |
This does not show that attention can never guide pruning. Consider a prompt that mentions France near the beginning and asks about its capital much later. That early token may still matter even after it falls outside a recent window. An attention-guided policy could help if its score correctly identifies and retains that connection. This is only an example, not a result from this experiment. The measured result is that this simple cumulative-attention rule was weaker than recency for this model and continuous-text workload.
Bounded windows: memory and decode-time impact
The full cache grew by 28 KiB for every original token because every layer must retain key and value vectors for that token. Once a bounded window reached its budget, its live memory stopped growing because each new entry replaced an old one. The model also avoided reading the complete history during every later attention step, so decode time remained much more stable as the original context grew.
| cache at 120,000-token context | live KV memory | decode speed | speedup |
|---|---|---|---|
| full history | 3,281.6 MiB | 4.26 tok/s | control |
| 4,096-token window | 112 MiB | 28.27 tok/s | 6.63 times faster |
| 2,048-token window | 56 MiB | 28.62 tok/s | 6.71 times faster |
| 512-token window | 14 MiB | 29.17 tok/s | 6.84 times faster |
The full cache became slower as context grew. Each bounded cache remained near 28 to 29 tokens per second after reaching its budget.
Pruning had a small cost at short contexts because selecting and copying cache entries also takes time. At a 4,096-token context, the 4,096-token window was 3.1% slower than the full cache. By 16,384 tokens, every bounded cache was faster because the saved attention work exceeded that overhead.
Conclusion
KV-cache pruning had a useful middle region in this experiment. A recent window removed one third of the cache with almost no measured quality change. A 2,048-token budget saved two thirds of memory for a modest quality cost. Smaller budgets saved more memory and speed, but perplexity worsened quickly.
The practical choice depends on whether quality, maximum context, or decode speed matters most. These numbers come from one model and one WikiText-2 slice. The long-context speed test used synthetic KV values, which are valid for runtime and memory but not for measuring model quality. A production ring, paged, or static cache could also reduce the copying overhead measured here. The prototype selected retained entries and copied them into a new tensor. A ring or static cache can reserve a fixed region and overwrite old entries in place, while a paged cache manages entries in reusable blocks. Both approaches can avoid much of that repeated copying.
GitHub repository
The complete report, raw measurements, benchmark code, and reproduction instructions are available in the LLM Inference Optimization repository.