module 11
Perceptron, MLP and Backpropagation
The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
Frank Rosenblatt
Proposes a machine that learns to recognise patterns by adjusting connection strengths whenever it makes a mistake. This single artificial neuron is the ancestor of every neural network.
Learning Representations by Back-Propagating Errors
David E. Rumelhart et al.
Shows how to train networks with hidden layers: send the error backwards through the network and nudge every weight in the direction that reduces it. The hidden units then learn useful features on their own.
the perceptron: one artificial neuron
Weigh the inputs, add them up, and answer yes if the total is positive. Rosenblatt's learning rule only acts on mistakes:
Each mistake turns the line a little toward the point it got wrong. If a straight line can separate the classes, this always finishes after a limited number of mistakes.
The limit: one neuron draws one straight line. XOR (two opposite corners are one class, the other two corners the other) can't be split by any single line, so the perceptron never settles.
stacking layers
Put a layer of neurons between the input and the output, and XOR becomes easy. The middle (hidden) neurons build their own features, and the output combines them.
The hard question was how to train the hidden layer. When the output is wrong, which hidden weights are to blame, and by how much?
backpropagation
Rumelhart, Hinton and Williams's answer: send the error backwards, layer by
layer, with the chain rule. Each layer z = x·W + b, a = f(z) does three things:
- turn the error coming from above into the error before the activation:
δ = error × f'(z) - work out its own weight gradient:
xᵀ · δ - pass the error further down:
δ · Wᵀ
One backward pass costs about the same as one forward pass and gives the gradient of every weight. It's the same backward idea as autograd (module 0).
The paper's real point was what this makes possible: the hidden neurons learn useful features on their own that nobody designed. Watch the boundary bend as training goes on:
the network learns the spiral
● filled class · ○ hollow class · darker background = more confident it's the filled class
how the code works
| idea | where |
|---|---|
| the perceptron rule | Perceptron.fit |
weight gradient xᵀδ, error passed down δWᵀ |
Dense.backward in layers.py |
δ = error × f'(z) |
ActivationLayer.backward |
| go backwards through every layer | Sequential.backward |
| the error at the output | for sigmoid or softmax with cross-entropy it's simply p − y |
| the training loop | Trainer.fit |
Each layer saves what its backward pass needs (its input, its value before the
activation) during the forward pass. So the whole network is just a list of layers,
and the CNN reuses the same Sequential and Trainer.
what the demo checks
Run uv run python examples/11_mlp.py.
- The perceptron finishes on separable data after 10 updates, well inside the theoretical limit of 356.
- It never settles on XOR. One hidden layer of 4 neurons solves it exactly.
- Every layer's backward pass matches a numerical estimate to about 10⁻⁹.
- The hand-written backpropagation gives exactly the same gradients as the autograd engine from module 0.
- On a spiral: a straight-line model gets 0.651, one hidden layer of 8 neurons 0.907, of 32 neurons 0.995, and two layers 0.998.
when it goes wrong
- The perceptron on XOR: it keeps cycling forever.
- Too few hidden neurons: the boundary can't bend enough.
- A wrong gradient: the classic bug. Always check against a numerical estimate.