LLM inference capacity planning: memory, load and latency

Describe what the application does, when requests arrive and how long the user can wait. An overnight summarisation job can tolerate a queue that an interactive assistant cannot. Record requests per interval, input length, expected output, completion deadline and acceptable error level. Separate normal traffic from known peaks, such as shift changes or scheduled batch submission.

Use authorised representative documents or equivalent synthetic inputs. Character length does not uniquely determine token count: the tokeniser and content matter. Include instructions, retrieved documents and conversation history that will actually reach the model. Model selection and capacity planning should inform each other because changing the model can change both quality and resource use.

A first approximation is weight memory = parameter count × bytes per parameter. A hypothetical dense model with eight billion parameters at two bytes per parameter needs about 16 decimal GB, or 14.9 GiB, for weights alone. This is neither a GPU recommendation nor a total memory estimate. Caches, activations, buffers, serving state and format overhead still need allowance.

At four bits per parameter, the idealised weight calculation becomes 4 decimal GB. Quantisation metadata and implementation overhead mean this is not the final memory footprint. It also does not imply four times the speed or unchanged quality. Check support and measure the exact candidate. For mixture-of-experts models, active parameters per token and resident weight memory are different quantities.

The KV cache retains attention states for reuse during generation. Its size cannot be inferred from total parameter count alone: layers, KV heads, dimensions and resident tokens matter. Check the vLLM cache configuration for the selected engine version. A configuration setting is not a performance guarantee.

One hundred registered users do not imply one hundred simultaneous generations. Observe arrival rate and time spent in the system. As a consistency check under stable conditions, average concurrency equals arrival rate multiplied by average time in the system. At two requests per second and six seconds per request, the average would be twelve requests. This arithmetic example is not a capacity test or a peak estimate.

A growing queue makes planning around that average inadequate. Test bursts and observe whether the system returns to its normal state within the accepted deadline. Bound input size, output size and outstanding requests. If exceptionally long contexts are permitted, test their effect on other users and decide whether they require a separate queue or quota.

For a streaming interface, time to first token describes the initial wait; subsequent generation speed affects completion time. For a workflow requiring valid JSON, the first token may provide little value: completing and validating the object matters. Include authentication, document retrieval, network time and validation, rather than measuring only the inference engine.

Report the median and high percentiles alongside sample size and offered load. An average can hide unacceptable waits. Distinguish failed requests from slow requests, and generated responses from accepted tasks. vLLM optimisation guidance describes configuration-dependent memory and performance choices. Repeat relevant measurements after changing those settings instead of assuming previous results still apply.

Pin the model version, engine, precision, hardware and limits. Prepare normal, peak and long-context profiles. Warm the service, run long enough to observe queue behaviour and record offered, completed, rejected and quality-accepted requests. Increase load in stages and retain the results of unsuccessful stages. Record whether a test was limited by the client generator as well as the serving system.

Then exercise recovery: restart a process, interrupt a dependency and reduce available capacity where the architecture permits. Do not claim redundancy without measuring service behaviour with a component unavailable. Derive headroom for variation and maintenance from these observations. The nominal throughput of a group of accelerators does not establish available capacity during an incident.

State which configuration passed, against which request distribution and acceptance criteria. Include maximum tested load, the point where the queue begins to grow and behaviour when limits are exceeded. If no configuration passes, reduce context or scope, change the model or reconsider the business deadline. Merely increasing a timeout does not resolve an overloaded service.

Carry the configuration into the pilot evaluation, its cost into the economic model, and thresholds into the operating plan. Private inference services begin by defining these conditions before committing to a platform. Useful evidence states both the operating range demonstrated and the scenarios that remain untested.