Before buying a GPU or a server for on-premises AI, the first question is always the same: will the model fit in memory? It is not a trivial question — the answer depends on more factors than it first appears, and getting it wrong means an underused server or, worse, one that cannot even load the model you need.
The three components that occupy VRAM
A GPU’s inference memory is divided into three blocks:
- The model weights. This is the largest and easiest component to estimate: it depends on the number of parameters and the numerical precision used to load them.
- The KV cache (key-value cache). It grows with the context (input + output tokens) and the number of concurrent requests. For models with long contexts, it can exceed the size of the model itself.
- Execution workspace (activations and inference engine buffers). This is usually the smallest component, but it is sensible to allow a 10–15% margin above the theoretical calculation.
The basic formula for model weights
VRAM (weights) ≈ number of parameters × bytes per parameter
The bytes per parameter depend on the precision:
| Precision | Bytes per parameter | 7B weights |
|---|---|---|
| FP16 / BF16 | 2 | ≈14 GB |
| INT8 | 1 | ≈7 GB |
| INT4 | 0.5 | ≈3.5 GB |
This explains why quantization is the first lever to adjust when a model “does not fit”: moving from FP16 to INT4 can reduce the VRAM requirement to one quarter, in exchange for a quality loss that should be measured for each use case (we cover this in detail in our quantization guide).
Why the KV cache changes the whole calculation
A common mistake is to size hardware only for the model and forget the KV cache. In production, with multiple concurrent requests and long contexts (documents and conversation histories), the KV cache can become the dominant memory component, exceeding the model itself.
The KV cache depends on context length, simultaneous sequences and architecture: layers, KV heads and head dimension. The precision used for the cache also matters.
We examine this in more detail in our article about the KV cache, because it is the variable that most often surprises people sizing a server for the first time.
A practical example
Consider weights for an 8-billion-parameter model in FP16, with 32 layers, 8 KV heads and a head dimension of 128. For 20 simultaneous sequences of 4,096 tokens, with 16-bit KV and no prefix sharing:
- Theoretical weights: ≈16 GB.
- KV cache: 2 × 32 × 8 × 128 × 4,096 × 20 × 2 bytes = ≈21.47 GB.
- Indicative 15% reserve on weights and KV: ≈5.62 GB.
Total for this example: ≈43.1 GB of VRAM.
These are decimal GB and explicit assumptions, not a measurement. The engine, its buffers and metadata may require more memory. A 24 GB GPU does not cover this complete scenario; quantization, limiting the context or distributing the workload changes the result.
From theory to your own scenario
These calculations are a useful approximation for understanding the problem, but each model has a different architecture (number of layers, attention heads and attention type) that affects the actual result. To avoid calculating everything by hand, you can use the servidores.tech LLM Calculator, which applies this same methodology using each model’s specific data and returns a report showing the available memory margin.