Deep learning basics: how a model learns a function from data
Neurons, layers, loss, gradients and backpropagation. Teach a small network XOR and see, briefly, how deep learning learns.
Deep learning fits the weights of a layered function to data by repeating predict, loss, gradient and update. LLMs are built on the same mechanism.
What deep learning is
Deep learning is a way to learn a function that turns inputs into outputs, from data. Instead of people writing rules, you build a function that outputs "what is in the picture" or "the next word" by showing it many examples. The function is simple computations (neurons) arranged in layers and stacked many deep. That stacking is where the name comes from.
What one neuron does
A neuron multiplies each input by its own weight, adds them up, adds a bias, and passes the result through an activation function. That is all.
- Weights and biases are the numbers decided by learning (the parameters)
- The activation function (ReLU, tanh, sigmoid and others) puts a "bend" into the computation. Without it, any number of layers would still be a straight-line function
- Stacking layers lets each layer build more complex features out of the previous layer's outputs
When people say an LLM has "billions of parameters", they mean the number of these weights and biases.
How learning works
Learning repeats four steps.
- Predict: compute outputs from inputs with the current weights (the forward pass)
- Loss: turn the gap between prediction and answer into one number
- Gradient: work out which way, and how far, to move each weight to reduce the loss. It is computed from the output side back toward the input, so it is called backpropagation
- Update: move the weights a little against the gradient (gradient descent). How far to move is the learning rate
A small example: learning XOR
XOR outputs 1 only when the two inputs differ. One straight line cannot separate it, so it needs a hidden layer. Here are the four steps above, written plainly with only NumPy (the output is from actually running it).
import numpy as np
# Data: XOR (1 only when the two inputs differ). One straight line cannot separate it
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)
# Model: 2 inputs → 8 hidden (tanh) → 1 output (sigmoid)
rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, 1, (2, 8)), np.zeros(8)
W2, b2 = rng.normal(0, 1, (8, 1)), np.zeros(1)
sigmoid = lambda z: 1 / (1 + np.exp(-z))
lr = 0.5 # learning rate: how far to move the weights each step
for epoch in range(2001):
# 1. Predict (forward pass)
h = np.tanh(X @ W1 + b1)
p = sigmoid(h @ W2 + b2)
# 2. Loss: how far from the answers (cross-entropy)
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
# 3. Gradients (backpropagation): which way to move each weight to reduce the loss
dz2 = (p - y) / len(X)
dW2, db2 = h.T @ dz2, dz2.sum(0)
dz1 = (dz2 @ W2.T) * (1 - h ** 2)
dW1, db1 = X.T @ dz1, dz1.sum(0)
# 4. Update: move a little against the gradient
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
if epoch % 500 == 0:
print(f"epoch {epoch:4d} loss {loss:.4f}")
print("prediction:", p.round(3).ravel(), "→", (p > 0.5).astype(int).ravel())epoch 0 loss 0.7414
epoch 500 loss 0.0068
epoch 1000 loss 0.0029
epoch 1500 loss 0.0018
epoch 2000 loss 0.0013
prediction: [0.001 0.998 0.999 0.002] → [0 1 1 0]The loss goes down, and it answers all four inputs correctly. With only 4 hidden units, some random seeds got stuck in a dip on the way (a local minimum) and failed to solve it (1 out of 20 in my trials). Widening the layer a little made it stable.
In practice a library such as PyTorch takes care of computing gradients (automatic differentiation) and running on GPUs. What you write is the shape of the model, the loss, and the training loop.
Common terms
| Term | Meaning |
|---|---|
| Epoch | One pass through all the training data |
| Batch | The chunk of data used for one update |
| Learning rate | How far each update moves the weights. Too large diverges; too small barely moves |
| Validation data | Data held out from training, used to check |
| Overfitting | Fitting the training data so closely that new data goes wrong |
| Inference | Only predicting, with trained weights |
Preventing overfitting
- Split the data: separate training and validation data, and judge by validation results
- Stop early: stop training when the validation loss stops falling
- Add data, or shrink the model: check the model is not too large for the data
How this connects to LLMs
LLMs work the same way. They build their layers with the Transformer and learn "predict the next token" from enormous amounts of text. That continues in How LLMs work and Inside the Transformer.