One of the most expensive architectural decisions to reverse in an on-premises AI deployment is starting with the wrong topology: oversizing with an unnecessary cluster, or falling short with a single server that cannot handle actual demand. These are the criteria that should guide the decision.

When a single server is enough

One server with one or more GPUs comfortably covers most initial on-premises AI deployments when:

  • The model fits into the available VRAM with a reasonable margin (see our guide to how much VRAM an LLM needs).
  • Expected concurrency is moderate and predictable — an internal team or a departmental use case.
  • The business can tolerate occasional service interruptions for maintenance or restarts.
  • The available budget and operational capacity favour simplicity over redundancy.

The advantage of this topology goes beyond hardware cost: it is also much simpler to operate, monitor and maintain, which matters especially for a small systems team.

When to move to a cluster

A cluster (several GPU nodes working together) begins to make sense when one of these factors appears:

  1. Concurrency exceeds a single node’s capacity. If the number of concurrent requests creates an unacceptable queue, scaling out distributes the load across nodes.
  2. High availability is a business requirement. If the AI service is critical (customer support or production processes), a single server is a single point of failure. Fault tolerance requires replicas, spare capacity and a tested failover strategy; adding nodes alone is insufficient.
  3. Models cannot fit in the GPUs of a single server and must be distributed across multiple nodes. Using several GPUs within the same server does not, by itself, require a cluster.
  4. Demand is expected to grow in the short or medium term, so designing for scale from the outset avoids a disruptive migration later.

The real cost of a cluster goes beyond hardware

Moving from one node to several does more than multiply GPU costs: it adds orchestration complexity (load balancing between nodes and state synchronisation), higher-bandwidth networking between nodes when the model is partitioned, and a greater operational workload (distributed monitoring, coordinated updates, backups and continuity at cluster level, not just machine level). This is the part of total cost most often underestimated during planning — we cover it in detail in our guide to on-premises vs cloud AI for businesses.

How to decide without oversizing

The practical recommendation is to start with a single server whenever the VRAM and concurrency calculations allow it, while designing the architecture so that a future cluster is an expansion rather than a rebuild — for example, by choosing an inference engine that supports distributed deployment from the outset, even if that capability is not used yet.

Before deciding, calculate how much VRAM and concurrency headroom your actual use case needs with the LLM Calculator — these figures determine whether a single server is viable or whether it makes sense to plan directly for a cluster.