How LLMs work: predicting the next token

What an LLM does

A large language model (LLM) is a language model with many parameters, trained on vast amounts of text. In pretraining, the foundation, it learns to predict the next word from the context so far in a document. A model that has only been pretrained isn’t good at answering questions or following instructions. So it is refined with fine-tuning (further training on additional data) and RLHF (training to match human preferences) to make it a usable assistant. Claude is built this way (per the official glossary). This article is based on the official documentation as of October 2026.

What follows is a simplification for understanding. We do not cover the internal structure of real models here.

Tokens: a unit that is neither a character nor a word

A model handles text not as characters or words themselves but as tokens. A token can correspond to a word, part of a word, a character, or a byte, and how many characters one token represents depends on the model and the language. According to the official glossary, Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text than earlier models. Exact counts come from the token counting API with the model you plan to use. We can’t count tokens in a local calculation, so this article gives no concrete token counts for Japanese or English.

• Pricing is based on input and output token counts
• The size of the context window is also counted in tokens
• The same content produces different token counts in different languages

Choosing the next word by probability

An LLM takes the tokens so far as input, computes “the probability of each possible next token,” and picks one. It appends that token and computes the next one again. Repeating this produces text. To check just the shape of it, I wrote a toy model that predicts the next word from only the previous word. It counts which words followed which in a short text and turns the counts into probabilities. A real LLM uses the whole context, not only the previous word. The “training” here is just counting. You can switch between the JavaScript and Python versions of the code with the tabs. Both use the same random-number formula, so the output is identical (we ran them on Node.js 26.1.0 and Python 3.11.5).

// A toy language model that predicts the next word from only the previous word.
const corpus = `
the model reads the text and predicts the next token .
the model predicts the next word from the context .
the context holds the text the model can see .
the answer looks plausible and the answer can be wrong .
the next token is chosen from a list of probabilities .
`.trim().split(/\s+/);

// 1. Training: count how many times each word followed each other word
const counts = new Map();
for (let i = 0; i < corpus.length - 1; i++) {
  const [a, b] = [corpus[i], corpus[i + 1]];
  if (!counts.has(a)) counts.set(a, new Map());
  counts.get(a).set(b, (counts.get(a).get(b) ?? 0) + 1);
}

// 2. Probabilities: turn counts into probabilities with a temperature (lower temperature concentrates on the most frequent words)
function probabilities(word, temperature = 1) {
  const next = [...counts.get(word)];
  const weights = next.map(([, n]) => Math.pow(n, 1 / temperature));
  const total = weights.reduce((s, w) => s + w, 0);
  return next.map(([w], i) => [w, weights[i] / total]);
}

// 3. Generation: draw the next word by probability. The random seed is fixed, so every run gives the same result
function rng(seed) {
  return () => {
    seed = (seed + 0x6D2B79F5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
function generate(start, temperature, seed, length = 12) {
  const random = rng(seed);
  const out = [start];
  while (out.length < length && counts.has(out.at(-1))) {
    let r = random();
    for (const [w, p] of probabilities(out.at(-1), temperature)) {
      if ((r -= p) <= 0) { out.push(w); break; }
    }
  }
  return out.join(' ');
}

for (const t of [0.5, 1, 2]) {
  console.log(`Temperature ${t}:`, probabilities('the', t).map(([w, p]) => `${w} ${(p * 100).toFixed(1)}%`).join(', '));
}
// How often is the most frequent word after "the" (model or next) chosen, over 2000 draws?
for (const t of [0.2, 1, 3]) {
  const random = rng(42);
  let top = 0;
  for (let i = 0; i < 2000; i++) {
    let r = random();
    for (const [w, p] of probabilities('the', t)) {
      if ((r -= p) <= 0) {
        if (w === 'model' || w === 'next') top++;
        break;
      }
    }
  }
  console.log(`Temperature ${t}: share of draws that picked a most-frequent word ${(top / 20).toFixed(2)}%`);
}
console.log('Example (temperature 0.2):', generate('the', 0.2, 1));
console.log('Example (temperature 3):', generate('the', 3, 1));

# A toy language model that predicts the next word from only the previous word (the same calculation as toy.mjs, in Python).
corpus = """
the model reads the text and predicts the next token .
the model predicts the next word from the context .
the context holds the text the model can see .
the answer looks plausible and the answer can be wrong .
the next token is chosen from a list of probabilities .
""".split()

# 1. Training: count how many times each word followed each other word
counts = {}
for a, b in zip(corpus, corpus[1:]):
    counts.setdefault(a, {})
    counts[a][b] = counts[a].get(b, 0) + 1


# 2. Probabilities: turn counts into probabilities with a temperature (lower temperature concentrates on the most frequent words)
def probabilities(word, temperature=1):
    words = list(counts[word].items())
    weights = [n ** (1 / temperature) for _, n in words]
    total = sum(weights)
    return [(w, weight / total) for (w, _), weight in zip(words, weights)]


# 3. Generation: draw the next word by probability. The random seed is fixed, so every run gives the same result
def rng(seed):
    state = seed & 0xFFFFFFFF

    def random():
        nonlocal state
        state = (state + 0x6D2B79F5) & 0xFFFFFFFF
        t = ((state ^ (state >> 15)) * (1 | state)) & 0xFFFFFFFF
        t = ((t + (((t ^ (t >> 7)) * (61 | t)) & 0xFFFFFFFF)) & 0xFFFFFFFF) ^ t
        return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296

    return random


def generate(start, temperature, seed, length=12):
    random = rng(seed)
    out = [start]
    while len(out) < length and out[-1] in counts:
        r = random()
        for w, p in probabilities(out[-1], temperature):
            r -= p
            if r <= 0:
                out.append(w)
                break
    return ' '.join(out)


for t in [0.5, 1, 2]:
    print(f'Temperature {t}:', ', '.join(f'{w} {p * 100:.1f}%' for w, p in probabilities('the', t)))

# How often is the most frequent word after "the" (model or next) chosen, over 2000 draws?
for t in [0.2, 1, 3]:
    random = rng(42)
    top = 0
    for _ in range(2000):
        r = random()
        for w, p in probabilities('the', t):
            r -= p
            if r <= 0:
                if w in ('model', 'next'):
                    top += 1
                break
    print(f'Temperature {t}: share of draws that picked a most-frequent word {top / 20:.2f}%')
print('Example (temperature 0.2):', generate('the', 0.2, 1))
print('Example (temperature 3):', generate('the', 3, 1))

Temperature: adjusting how skewed the choice is

Running the code above gives the following (we ran it on Node.js 26.1.0; the random seed is fixed, so every run gives the same result).

Temperature 0.5: model 30.0%, text 13.3%, next 30.0%, context 13.3%, answer 13.3%
Temperature 1: model 25.0%, text 16.7%, next 25.0%, context 16.7%, answer 16.7%
Temperature 2: model 22.5%, text 18.4%, next 22.5%, context 18.4%, answer 18.4%
Temperature 0.2: share of draws that picked a most-frequent word 83.30%
Temperature 1: share of draws that picked a most-frequent word 51.25%
Temperature 3: share of draws that picked a most-frequent word 43.85%
Example (temperature 0.2): the next token is chosen from the next token . the text
Example (temperature 3): the context . the answer can see . the next word from

After “the,” model and next each appear 3 times and the rest 2 times each. At temperature 1, the probabilities follow the counts (25% and 16.7%). Lowering the temperature concentrates probability on the frequent words (30% at temperature 0.5). Raising it narrows the gap (22.5% at temperature 2). Over 2000 draws, the share that picked a most-frequent word was 83.30% at temperature 0.2, 51.25% at temperature 1 and 43.85% at temperature 3, close to the theoretical 83.5%, 50.0% and 43.3%. The official glossary says the same thing: higher temperature gives more varied output, and lower temperature sticks to the most likely words. In the generated examples, temperature 0.2 nearly retraced the original text, while temperature 3 produced a combination that isn’t in it (the answer can see).

Even with temperature set to 0, an API’s results aren’t guaranteed to be identical. The same input can produce different outputs across calls (official glossary).

Move the temperature and see

Move the slider and the probabilities of the word after “the” are recomputed on the spot, using the same formula as the toy model in the article. Roll the dice to see how it actually comes out. The more you roll, the closer the results get to the theory.

The context window: working memory

The context window is all the text a model can reference when producing a response; you can think of it as “working memory.” It is separate from the large corpus the model was trained on. It holds the system prompt, every message so far, tool results, and images or documents, and the output the model generates counts too. As the conversation advances, all earlier turns pile up. The size depends on the model: in the documentation as of October 2026, many current models have 1 million tokens and the others 200 thousand.

• Bigger isn’t automatically better. As tokens grow, accuracy and recall degrade, a phenomenon called “context rot” (official)
• If the input alone exceeds the limit, the API returns a 400 error (prompt is too long)
• On Claude 4.5 and later models, a request is accepted even if input plus max_tokens exceeds the limit, and generation stops when it reaches the limit (stop_reason: model_context_window_exceeded)

This may be why instructions are followed less reliably after a long session in Claude Code. The remedy is to keep only what you need in context (start a new session, move investigation into a separate context).

Why it gets things wrong

An LLM isn’t looking up facts to answer; it chooses tokens that plausibly follow the context. So it can produce text that sounds right but is factually wrong (hallucination). The official documentation says this can happen even with the most advanced models. Even the toy model produced a combination that isn’t in its source text, exactly as the probabilities allow. By design, it can happen. The mitigations the official guide lists are:

• Tell it plainly that it may say “I don’t know”
• For long documents, have it extract word-for-word quotes first, then answer based only on those quotes
• Have it find a supporting quote for each claim, and retract the claim if it can’t
• Ask the same question several times and look for inconsistencies (best-of-N)
• Instruct it to use only the information in the documents you provide

These reduce hallucinations significantly but don’t eliminate them. Always verify important information yourself (official).

Letting the mechanism decide how you use it

The mechanism so far leads to a few working rules.

• Put the information it needs into the context. Don’t assume the model already knows it (training data and context are separate)
• Whatever you paste enters the context. Remove secret keys and personal information before sharing
• The same request can get different answers. Check the result every time
• Split a long conversation before quality drops

The Claude Code articles (getting started, CLAUDE.md, permissions, workflows) cover concrete usage.

How the toy model differs from a real LLM

The shape is the same as in a real LLM: choose the next token by probability, and shift the skew with temperature. What differs is how the probabilities are computed.

The inside of the part that computes these probabilities is covered, with figures and code, in A look inside the Transformer.