Memory methodology · v0.3

Changes since v0.1

Implemented formulas

P is total parameters, U unquantized parameters, L layers, H hidden dimension, S context, B microbatch and V vocabulary. Parameter counts entered in billions are multiplied by 10⁹. All calculated memory is converted using bytes / 10⁹.

• Weights: (P − U) × bits/8 × (1 + metadata) + U × 2 + U32 × 2, when quantized, where U32 is the subset retained in FP32. At 4 bits, GPT-OSS applies a minimum of 6.25% for scales. At 16 bits, U32 remains at 32 bits; at 32 bits, everything uses 32 bits. The metadata percentage is not added at 16/32 bits.
• Qwen embeddings: U ≈ 2 × V × H, for untied embeddings and output. BF16 is assumed, but a particular artifact may quantize them. This can be changed for a custom model.
• TP: divides weights. Local KV heads = max(1, ceil(KV heads / TP)). When TP exceeds the number of KV heads, they are replicated. Compressed MLA is budgeted as replicated per TP rank.
• PP: layers in the largest stage = ceil(L/PP). Body weights per GPU = body × stage_layers/L/TP; U is budgeted in full in each stage, divided by TP. This is a conservative placement bound, not a tensor-by-tensor assignment.
• KV MHA/GQA/MQA: 2 × stage_layers × local_KV_heads × head_dimension × resident_tokens × KV_bytes. MHA uses the Q heads and MQA uses one.
• KV MLA: stage_layers × (latent + RoPE) × resident_tokens × KV_bytes. Only valid for a backend that stores the compressed representation.
• Residency: replicas = machines × GPUs/(TP × PP); requests/replica = ceil(users/replicas); active = min(batch, requests/replica). The selected policy reserves KV for all requests or only active requests.
• Blocks and prefix: shared prefix rounded down to complete blocks; private portion rounded up per sequence. Window: min(S, window) only in the declared layers. PP uses the layer combination with the greatest KV as its budget.
• Activations: 2 × K × tokens × H. During inference, tokens = min(active × S, prefill budget), without multiplying by L because memory is reused between layers. During training, tokens = B × S and this is multiplied by the stage's layer count. These activations are not divided by TP: a conservative budget without sequence parallelism.
• Logits: 4 × V × active during inference (last token only); 4 × V × B × S during training. These are budgeted without vocabulary partitioning. Without efficient attention, an approximate matrix of 2 × Q_heads/TP × B × S² bytes is added, multiplied by stage layers during training.
• Training: per trainable parameter, 8 bytes for AdamW, 2 or 4 for gradients and an optional 4 for a master copy. Adapters add their own BF16 weights. The LoRA base remains in BF16 and QLoRA in 4 bits with configurable metadata.
• Communication: two BF16 buffers of tokens × H for each active TP/PP dimension. During inference, tokens are limited to the prefill budget. This is a project budget, indirectly editable through the chunk size, not a published NCCL calibration.
• Total per GPU: the components above + offload staging + percentage margin + engine context. Available = nominal capacity × (1 − reserve). The state is determined by comparing these values per GPU.
• Offload: adds staging for one layer's body per GPU. Offload cannot exceed the quantized body weights or, summed across all GPUs, 80% of the machine's RAM. This is a capacity limit without simulation of transfers, initial loading or actual layer granularity.

Assumptions that must not be interpreted as measured data

Project defaults: K=20, engine context=1.073741824 GB/GPU, margin=15%, reserve=10%, quantized metadata=5%, KV blocks=16 tokens and prefill=2048 tokens. The margin does not replace explicit components or represent a measurement.

The engine, interface and report use decimal GB: 1 GB = 1,000,000,000 bytes. For example, 8 billion FP16 parameters occupy 16 GB before other components. Catalogue capacities retain the published nominal GB without reinterpreting them as binary units. This is a planning budget, not a measurement of physical bytes: confirm the memory visible to the engine. Previous binary engine and OS budgets are converted to GB while preserving their byte counts. Bandwidth is in GB/s and disks in decimal TB (1 TB = 1,000 GB).

The approximate 0.04 multiplier is not applied to all MLA models: their dimensions are used. MoE VRAM is not reduced using a probabilistic curve of active experts either: all weights remain resident. Selecting a GPU does not guarantee support for kernels or formats.

Limits remain: no linear/DSA/CSA attention, explicit multimodal encoders, ZeRO/FSDP, checkpointing, inter-node parallelism, speculative decoding or measured TPS/TTFT prediction. The indicative inference range is explained below. The network card does not itself change the model's byte count.

Visual panel and indicative speed (v0.3)

The circle represents the budget of the most heavily loaded GPU divided by its usable VRAM, after reservation. Up to 60%: ample headroom; 60–85%: headroom; 85–100%: tight; above 100%: does not fit without changes or RAM offload. These are project planning categories, not manufacturer thresholds. When capacity is exceeded, the circle fills and the percentage retains its value above 100%; the text shows the missing GB.

Speed follows a compute and bandwidth roofline approach using planning formulas and assumptions. It is offered only for preloaded dense models with standard full-context attention, efficient attention, TP=PP=1 and sufficient memory without offload. MoE, MLA, sliding windows and special dimensions have no speed estimate. No figures are assigned to training, custom models, GB300 NVL72 or uncalibrated TP/PP communication.

Per occupied replica: B = min(batch, ceil(users/replicas)). Each decode step reads all weights plus the KV of B sequences at the selected maximum context. The shared prefix is not deducted from KV reads. Approximate FLOPs = B × (2P + 4LHS). Step time = max(bytes/effective bandwidth, FLOPs/effective compute) / penalty + 0.002 seconds. For weights below 8 bits, the penalty is 0.92; INT4/FP8 arithmetic acceleration is not promised. Dense BF16 TFLOPS are used, without sparsity.

The conservative scenario uses 35% of bandwidth and 20% of compute; the favourable scenario uses 70% and 45%. These are illustrative assumptions, not percentiles, confidence intervals or measured results. Tokens/s per user = (B / requests per replica) / step time. If requests are queued, this is a long-term average with fair sharing, not a guaranteed continuous cadence. Balanced distribution between replicas and sufficient CPU input are assumed.

Prefill without queueing: max((2PN + 4LHN²)B/effective compute, weight bytes/effective bandwidth) + 0.002 s, with N limited to context. This approximates prompt processing, not end-to-end TTFT: it omits tokenization, model loading, network, scheduling, contention and waiting. The bar compares the conservative tokens/s bound with the entered target; it saturates at 2× the target while the number retains the full range.

CPU, RAM and network receive no arbitrary tokens/s multipliers: their actual effect requires measurement of the engine, topology, traffic and data supply. Adding replicas can serve more users; it does not automatically accelerate one request. Quantization and the backend must be supported by the GPU.

Figures marked as sparse in specifications are divided by two to obtain the dense reference. Decimal GB/s is not equated with GiB/s.

Expanded catalogue

311 entries reviewed: 224 calculable and 87 excluded because of insufficient data or support. Each model's own dimensions and limits are loaded; Qwen's architecture is not reused for other families. See scope, sources and limitations by model.

Verification

Automated numerical tests: a GQA example, mixed precision, MLA, residency, prefix, blocks, SWA, TP/PP, KV replication, queueing, AdamW, offload and rejection of invalid configurations. These do not replace measurement with a real LLM.