Running local LLMs seriously: big models and the KV cache

What this article covers

In the local LLM introduction you estimated the weights as parameters × bytes per parameter. For one person trying a model, that is enough. Once you run a big model, handle long text, or share it among several people, other factors start to matter. This article covers how to size them and how to choose tools.

Models and tools change quickly, so this article names no specific models and gives no speed numbers. The settings in the example are illustrative values.

Beyond the weights: the KV cache

When a model produces text one token at a time, it keeps in memory the intermediate results (the Key and Value) it computed for all the tokens so far, so it need not recompute them each time. This is the KV cache, and it grows with longer conversations and with more simultaneous users.

KV cache size = 2 (Key and Value) × layers × KV heads × head dimension × bytes per element × tokens × concurrent users

The number of layers and heads is in the model's config file or official page. The code below uses an illustrative setting (32 layers, 8 KV heads, dimension 128, 16-bit). The JavaScript and Python versions print the same output (checked by running both with Node.js and Python 3).

// KV cache = 2 (K and V) × layers × KV heads × head dim × bytes per element × tokens
// The layer count and the rest are illustrative values (not a real model's config).
const model = { layers: 32, kvHeads: 8, headDim: 128, bytes: 2 };   // 16-bit cache

const perToken = 2 * model.layers * model.kvHeads * model.headDim * model.bytes;
const GiB = (n) => n / 1024 ** 3;

console.log(`Per token: ${perToken / 1024} KiB`);
for (const tokens of [4096, 32768, 131072]) {
  for (const users of [1, 4]) {
    console.log(`${tokens} tokens, ${users} at once: ${GiB(perToken * tokens * users).toFixed(1)} GiB`);
  }
}

# KV cache = 2 (K and V) × layers × KV heads × head dim × bytes per element × tokens
# The layer count and the rest are illustrative values (not a real model's config).
model = {"layers": 32, "kv_heads": 8, "head_dim": 128, "bytes": 2}   # 16-bit cache

per_token = 2 * model["layers"] * model["kv_heads"] * model["head_dim"] * model["bytes"]
gib = lambda n: n / 1024 ** 3

print(f"Per token: {per_token // 1024} KiB")
for tokens in [4096, 32768, 131072]:
    for users in [1, 4]:
        print(f"{tokens} tokens, {users} at once: {gib(per_token * tokens * users):.1f} GiB")

Per token: 128 KiB
4096 tokens, 1 at once: 0.5 GiB
4096 tokens, 4 at once: 2.0 GiB
32768 tokens, 1 at once: 4.0 GiB
32768 tokens, 4 at once: 16.0 GiB
131072 tokens, 1 at once: 16.0 GiB
131072 tokens, 4 at once: 64.0 GiB

With these illustrative settings, a 4,000-token conversation needs only 0.5 GiB, but 131,072 tokens for four users needs 64 GiB on top of the weights. With long contexts on a big model, the cache, not the weights, can be the wall. The unit here is GiB (powers of 1024), slightly different from the GB (billions) in the introduction.

• Models with fewer KV heads (several heads sharing one Key and Value) have a smaller cache
• Some tools can also quantize the KV cache to shrink it, though quality may drop
• Just capping the context length at what you need can save a lot of memory

Running a model that is too big

For a model that does not fit on one GPU, there are several ways.

Sizing MoE (mixture of experts) models

An MoE model holds many “expert” parts and uses only some of them for each token. You must keep all the experts' weights in memory, so you size memory by the total parameter count. The computation per token uses only the active part, so it is lighter than a regular model of the same total size. Remember: memory by the total, compute by what is used.

What decides speed

Generating each token reads nearly all the weights from memory. So generation speed tends to depend on memory bandwidth (how much can be read per second) more than on raw compute. Smaller weights (quantized) mean less to read and faster output.

• Quantization helps speed as well as memory
• For the same weights, a machine with more bandwidth is faster. This is a separate condition from whether it fits
• The time to the first token grows with input length. The longer the document, the longer the wait

Serving several people: server tools

For one person, Ollama or llama.cpp is enough. For several simultaneous users, a server tool that batches concurrent requests fits better. vLLM is one example; it is built around batching requests efficiently and managing the KV cache well in memory. Check the official documentation for details.

Tools also differ in the model formats they accept. For example, llama.cpp uses GGUF, while GPU servers often use quantized formats such as AWQ or GPTQ, so check the format when you look for a model.

Checklist before serious use

• Do the weights (check with the estimator) plus the KV cache (longest context × concurrent users) fit in the memory?
• If you split across GPUs, is the link between them fast enough?
• Does your hardware's memory bandwidth give the speed you need? Would a smaller model or stronger quantization do?
• Have you decided the number of simultaneous users and the context length per user?
• Does the model's license allow your setup (internal use, commercial use, and so on)?
• Have you compared quality against a cloud model on your own task? Do not decide on cost alone
• Have you decided who handles updates, monitoring and failures?

With a cloud API, the provider handles all of this. Running it yourself pays off when you have a reason: data that cannot leave, high volume, or the need for fine control.