GPU, host and software
Engine 1.0-rfc2 · Reviewed: 26 September 2026. Memory in decimal GB; planning figures, not performance measurements.
GPU memory
- Weights. Total parameters are split into quantizable and non-quantizable parameters; embeddings are not counted twice. The model configuration's weight-sharing setting is preserved. For supported dense Llama, Mistral and Qwen2 models, the calculation identifies
(2 × layers + 1) × hidden dimensionRMSNorm parameters. Other architectures retain the catalogue's aggregate inventory without extrapolating that normalization count. - Weight metadata. The grouped profile adds scales and zero points of the specified size. The approximation
ceil(parameters / group)does not reproduce per-matrix padding or every kernel's auxiliary data. AWQ and GPTQ support different layouts; select the scale bytes and zero-point bits used by the actual file. The manual profile keeps the alternative percentage without adding it again. - Paged KV. Context is rounded up to the next block; only complete prefix blocks are shared. Data, scales and the block table are shown separately. vLLM uses 32-bit indices in its tables; the initial 8 bytes per entry is an editable, conservative generic budget. The per-tensor, per-head or grouped scaling policy must match the engine.
- CUDA and communication. Ampere 1.2 GB, Hopper 2.2 GB and Blackwell 3 GB are project budgets, not memory usage certified by NVIDIA. They can be replaced with a manual value. NCCL adds, per GPU,
channels × buffers per channel × buffer size; this is not multiplied by TP again. Tensor buffers are counted separately. The initial buffer is 4 MiB, converted to 0.004194304 GB. - Parallelism. TP must divide the query-head count; TP × PP must divide the number of GPUs in the machine. PP uses the stage with the most layers and reserves the unquantized block in every stage as a conservative bound. Mathematical validity does not certify support for every kernel or interconnect.
Sources: Transformers configuration, RMSNorm in Llama, AWQ in vLLM, MIT AWQ, KV tables, KV scales, NCCL.
Host per machine
| Resource | Budget | How to interpret it |
|---|---|---|
| RAM | OS + weight copy + optional queued KV + transfer + sessions + software | Initially: 32 GB for the OS, 2 GB/GPU for transfers and 25 MB/user. The weight copy and offload share the same reservation; with unified memory, the GPU workload is not counted again. |
| Physical CPU | 4 + 2 × replicas + GPUs + ceil(users / 16) + software | A provisioning policy, not an equivalence between cores and tokens/s. Tokenization demand is also compared with the profile or measurement. |
| API network | Inbound: users × (prompt × bytes/token + payload) / arrival window. Outbound: users × target tokens/s × bytes/token. | For full duplex, the larger direction is compared, not their sum. One port per path; links are not aggregated automatically. Storage or fabric requirements are assessed separately. |
| Startup | Weights / min(read rate, transfer rate to GPU) + initialization | This is the ideal overlapped bound. The serial bound adds the two transfer times. Both exclude deserialization and compilation that have not been budgeted. |
Queued KV in RAM requires explicitly enabling the host reservation: being queued does not imply that the engine stores that cache. RAM below 1.5 × VRAM only produces a balance observation; it is not a universal requirement. Arithmetic division between sockets prompts an affinity review but does not identify NUMA nodes by itself.
Sources: vLLM engine options, CUDA transfers and memory, ThinkSystem performance tuning.
Software and bill of materials
The selector distinguishes licences, local reservations and external services. Licensing NVIDIA AI Enterprise does not itself consume VRAM. The optional NIM/Triton reservation is configured separately and added to the engine context only once. Management VMs are shown with their own resources; these are not subtracted from inference CPUs/GPUs.
OpenShift AI, ODF and their dependencies have node, storage and logical-CPU requirements. The SMT setting converts only these software requirements into a physical-core budget. It does not replace SNO, compact or distributed profiles that have not been modelled.
Verified ordering codes appear as P/N. CTO bases and feature codes are labelled differently. Disk capacities represent a documented format/capacity reference, not every possible backplane combination. If a P/N or licence scope has not been verified, the report says so explicitly. The list requires commercial and assembly validation before it becomes a quotation.
Sources for each licence: links in the PDF bill of materials and the JSON export. Each service's requirements and budgets are also included in the JSON.
Result states
- Feasible according to the estimate: memory and host budgets fit, and the conservative speed range meets the target.
- Conditional feasibility: metrics are missing, offload is used, context exceeds the specification, or software conditions remain unresolved.
- Insufficient configuration: a calculable capacity or a requirement of the selected profile is not met.
The result is an estimate. It does not certify full compatibility, commercial availability, observed performance or latency under load. The 30/60/90% scenarios express VRAM occupancy, not percentages of overall performance.