Local LLMs: running a model on your own machine

What a local LLM is

Running a local LLM means downloading the model's weights (the parameters learned in training) and doing inference on your own computer. A cloud API such as Claude's sends your input to the provider's servers and returns the result. With a local model, input and output stay on your machine. The mechanism is the same either way: choose the next token by probability, over and over (see How LLMs work for the basics). What differs is where the computation happens.

Available models and their performance change quickly, so this article names no specific models and gives no benchmark numbers. When you choose one, check the model's official page and its license.

Cloud vs. local

Neither is better in general; choose by use case. Local fits when you don't want sensitive documents to leave the machine, when you work without a network, or when you want to call a model many times while experimenting. Cloud is more realistic when you need the best quality or lack the hardware. Many people use both.

Estimating the memory you need

The first limit you hit locally is memory. The weights must fit in memory, and their size is roughly parameters × bytes per parameter. At 16 bits (2 bytes) per parameter, a 7-billion-parameter model (7B) takes about 14 GB. The code below prints the estimate for each size and precision. The JavaScript and Python versions print the same output (checked by running both with Node.js and Python 3).

// Weight memory in GB = parameters (billions) × bytes per parameter
const GB = (billions, bits) => (billions * 1e9 * bits / 8) / 1e9;

for (const size of [7, 14, 32, 70]) {
  const row = [16, 8, 4].map((bits) => `${bits}bit ${GB(size, bits).toFixed(1)} GB`);
  console.log(`${size}B:`, row.join(' / '));
}

# Weight memory in GB = parameters (billions) × bytes per parameter
def gb(billions, bits):
    return (billions * 1e9 * bits / 8) / 1e9

for size in [7, 14, 32, 70]:
    row = [f"{bits}bit {gb(size, bits):.1f} GB" for bits in [16, 8, 4]]
    print(f"{size}B:", " / ".join(row))

7B: 16bit 14.0 GB / 8bit 7.0 GB / 4bit 3.5 GB
14B: 16bit 28.0 GB / 8bit 14.0 GB / 4bit 7.0 GB
32B: 16bit 64.0 GB / 8bit 32.0 GB / 4bit 16.0 GB
70B: 16bit 140.0 GB / 8bit 70.0 GB / 4bit 35.0 GB

To try whether a model fits your memory, use the Local LLM Memory Estimator in the app. Pick the parameters, precision and memory to see the weights' size and a rough verdict.

These figures cover weights only. In practice you also need room for the cache that grows with conversation length (the KV cache), plus the OS and other apps, so leave headroom beyond the table.

Which "memory" this means depends on how you run the model.

• With a GPU (such as NVIDIA): video memory (VRAM). If the weights fit in VRAM, it runs fast. Some tools can spill the rest into main memory, but it becomes much slower
• On Apple Silicon: the CPU and GPU share one memory (unified memory). There is no separate video memory; ordinary memory serves the model too. The OS needs a share, so not all of it is available to the model
• On the CPU only: main memory. It works, but is slower than on a GPU

Shrinking it with quantization

As the table shows, fewer bits per parameter means proportionally less memory. That is quantization: you trade some precision to run the same model in less memory. Many published models come in versions quantized to around 4 bits.

• Fewer bits make the model lighter, but answer quality drops a little at a time
• A larger model at 4 bits is often said to beat a smaller one at high precision, but it depends on the model and task, so compare on your own machine
• Judge the quality difference on the tasks you plan to run, not on general numbers alone

Running a model with Ollama

The easy way to start is a tool like Ollama. It handles downloading models, running them in quantized formats, and serving a local API. Others include LM Studio (with a GUI) and llama.cpp, which many of these tools build on.

# Download a model and start chatting (pick <model-name> from the Ollama library)
ollama run <model-name>

# List the models you have downloaded
ollama list

Once running, Ollama listens for API requests on your own machine (localhost), and your programs query it over HTTP. Choose the first model by the size that fits your free memory according to the estimate above. Commands and API details can change, so check the official documentation.

Where it fits, and its limits

• Fits: summarizing or classifying sensitive documents, offline use, bulk drafting and reformatting, experiments to learn how models work
• Limits: it may be less capable than the largest models, long contexts and complex reasoning can be less stable, and speed depends on your hardware
• Caution: a local model still makes mistakes. Local or cloud, check the output before you rely on it

Checklist before you start

• Is a smaller model enough for the task?
• Do the weights plus headroom for the cache fit in free memory?
• Does the model's license allow your use (such as commercial use)?
• Do you have enough disk space (models run from a few GB to tens of GB)?
• Have you compared output quality on your own task?

For big models, several users or long contexts, see the follow-up, Running local LLMs seriously.