If you have ever sized a server for an LLM based only on the model’s size and then run out of memory in production, the KV cache was almost certainly responsible. It is the least intuitive variable in the VRAM calculation — and often the most decisive.

What the KV cache is

When a model generates text token by token, each new token needs to “remember” the attention information (the keys and values, hence KV) of all the previous tokens in the context. Instead of recalculating that information at each step — which is computationally very expensive — the inference engine stores it in memory: this is the KV cache.

Reusing keys and values avoids recalculating them at every step. This does not make the cost constant: attending to a context of 5,000 tokens is still more demanding than attending to one of 500. The cache also grows with the context.

What determines the KV cache size

Unlike model weights, which are fixed once the precision is chosen, the KV cache is dynamic and depends on:

  1. Context length (prompt + generated tokens). More context means more cache.
  2. The number of concurrent requests. Each active user maintains their own KV cache in parallel — ten simultaneous conversations of 4K tokens consume ten times as much cache as one.
  3. The model architecture: number of layers, number of attention heads, and whether it uses reduction techniques such as Grouped Query Attention (GQA) or Multi-Query Attention (MQA), which significantly reduce cache size compared with traditional Multi-Head Attention (MHA).

Why it causes surprises in production

In tests with one user and a short context, the KV cache is a small component compared with the model weights. The problem appears when scaling: with several concurrent requests and long contexts (complete documents or extensive conversation histories), the KV cache can exceed the size of the model weights themselves. It is the most common reason why a server that “worked well in testing” runs out of memory in production.

How to reduce its impact

There are several strategies, each with its own trade-off:

  • Choose models with GQA/MQA instead of full multi-head attention when the use case allows — this dramatically reduces the cache without changing weight precision.
  • Quantize the KV cache itself (not only the weights), a technique increasingly supported by modern inference engines.
  • Limit the effective context window when the full available history is unnecessary.
  • Size for the realistic worst case of concurrency, rather than the average — a usage peak without enough memory margin causes outages, not just slower responses.

How to include the KV cache in your sizing

This is precisely the component that can make two servers with the same GPU behave very differently depending on the model and usage pattern. We include it in the complete calculation in our guide to how much VRAM an LLM needs, and you can estimate it for your specific scenario — model, context and concurrent requests — in the LLM Calculator.