Trying LLMs on Colab: run a model on a free GPU

What you can do on Colab

Google Colab is a service where you write and run Python code in a notebook you open in the browser. Even the free plan lends you a machine with a GPU. Without a GPU in your own PC, you can still run an LLM and check the memory and quantization ideas hands-on.

This is not “local.” Colab's machines are in Google's cloud. The text you enter and the models you download run there. Do not put confidential documents or personal data in it. If you want the data to stay with you, run it on your own PC as in the local LLM introduction.

Good and poor uses:

Limits to know

The limits are in Colab's official FAQ. This article follows what I checked in October 2026.

• GPU types change: the GPUs available in the free tier change over time. A T4 is commonly said to be assigned often, but nothing is guaranteed
• Resources are not guaranteed: free usage limits fluctuate. When it is busy, you may get no GPU at all
• There is a time limit: a free notebook runs at most 12 hours, and it disconnects after a while of inactivity
• The runtime disappears: virtual machines are deleted when idle and have a maximum lifetime. The models you downloaded disappear with it. The notebook itself is saved in Google Drive

Some uses are not allowed (excerpt):

• Web services and other offerings unrelated to interactive compute (including exposing it to the outside)
• Connecting to remote proxies
• Remote control such as SSH, or bypassing the notebook to work mostly through a web UI (on the free tier this can be stopped without warning)
• Mining cryptocurrency, and using several accounts to get around limits

Some guides show how to use a tunneling service so that outside apps can reach Ollama on Colab. Judged against the FAQ, that use is likely to fall under the restrictions, so this article does not cover it. All calls are made from inside the same notebook.

The overall setup

Notebook cells run Python code, and an inference engine (Ollama or Transformers) puts the model weights on the GPU and produces answers. All of it runs inside a temporary virtual machine that Google provides. The model is downloaded from its source into that machine every time. The notebook itself is saved in Google Drive.

Preparation: attach a GPU

1. Create a new notebook
2. In the “Runtime” menu, choose “Change runtime type” and select GPU as the hardware accelerator
3. In the first cell, check that a GPU was assigned

%%bash
nvidia-smi

If a GPU name and memory size are shown, you have one. If not, recheck the runtime type, or try again after a while.

Method A: run Ollama inside the notebook

Install the Ollama you used in the local LLM introduction inside the notebook and run it.

%%bash
curl -fsSL https://ollama.com/install.sh | sh

import subprocess
import time
import urllib.request

# Start the server as a background process so the notebook is not blocked
server = subprocess.Popen(["ollama", "serve"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

for _ in range(30):  # wait up to 30 seconds for it to start
    try:
        urllib.request.urlopen("http://localhost:11434")
        break
    except Exception:
        time.sleep(1)

%%bash
# Pick <model-name> from the Ollama library. You download it again each time you reopen the notebook
ollama pull <model-name>

import json
import urllib.request


def chat(question, model="<model-name>"):
    body = {"model": model, "messages": [{"role": "user", "content": question}], "stream": False}
    request = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=300) as response:
        return json.load(response)["message"]["content"]


print(chat("What is the capital of Japan? Answer in one sentence."))

The install may fail with an error about a missing package. Follow the message to add what is needed, then try again. Install steps can change, so also check the official documentation. localhost means only the machine of this notebook. People outside cannot connect to it.

Method B: run it directly with Transformers

The other way is to load the model directly with Hugging Face's Transformers library. No server is needed; it all stays in Python. To fit even a GPU with small memory, load it quantized to 4 bits (bitsandbytes).

%%bash
pip install -q transformers accelerate bitsandbytes

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

name = "<model-name>"  # a model name on Hugging Face

# Load in 4 bits. Older GPUs such as the T4 may not support bfloat16, so use float16
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, quantization_config=quant, device_map="auto")

messages = [{"role": "user", "content": "What is the capital of Japan? Answer in one sentence."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Some models require you to accept their terms on Hugging Face and use an access token. Do not write the token in the notebook's code; store it in Colab's “Secrets” feature. That keeps it from leaking when you share the notebook.

Estimate the size of model that fits

The memory of the GPUs you can use for free is commonly said to be around 16 GB (the GPU type changes). The weights and the KV cache must fit in it. Check first with the memory estimator or the formula in the introduction.

• A 7-billion-parameter (7B) model loaded in 4 bits has about 3.5 GB of weights, with room to spare
• A 14B model in 4 bits is about 7 GB, and is likely to fit
• A 32B model in 4 bits is about 16 GB, which fills the memory and probably does not fit
• Long contexts and answering several questions at once grow the KV cache (see the follow-up article)

If you try to load a model that does not fit the GPU, it stops with an out-of-memory error. Then pick a smaller model or fewer bits.

Keeping your notebook tidy

When the runtime disappears, the downloaded model disappears with it. Decide these things ahead of time and you will not be stuck.

• Gather the cells that build the environment at the top of the notebook (GPU check, installs, starting the server). After reopening, run them from the top to be back where you were
• Basically download the model again each time. It takes time, so start with small models
• To keep results, save them outside the runtime, for example to Google Drive
• Before starting a long job, confirm a GPU is assigned

When it does not work

Checklist

• Have you kept confidential and personal data out of the notebook? (it runs in Google's cloud)
• Did you set the runtime type to GPU and confirm with nvidia-smi?
• Did you estimate that the chosen model fits in the GPU memory?
• Did you store the access token in Secrets rather than in the code?
• Are you avoiding any use that exposes it to the outside? (prohibited on Colab)
• Did you gather the environment-building cells, and decide where results are saved?