module 11

Perceptron, MLP and Backpropagation

paywalled

The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain

Frank Rosenblatt

Proposes a machine that learns to recognise patterns by adjusting connection strengths whenever it makes a mistake. This single artificial neuron is the ancestor of every neural network.

free pdf

Learning Representations by Back-Propagating Errors

David E. Rumelhart et al.

Shows how to train networks with hidden layers: send the error backwards through the network and nudge every weight in the direction that reduces it. The hidden units then learn useful features on their own.

the perceptron: one artificial neuron

Weigh the inputs, add them up, and answer yes if the total is positive. Rosenblatt's learning rule only acts on mistakes:

if wrong:w←w+η yixi,b←b+η yi\text{if wrong:}\quad w \leftarrow w + \eta\, y_i x_i,\qquad b \leftarrow b + \eta\, y_i

Each mistake turns the line a little toward the point it got wrong. If a straight line can separate the classes, this always finishes after a limited number of mistakes.

The limit: one neuron draws one straight line. XOR (two opposite corners are one class, the other two corners the other) can't be split by any single line, so the perceptron never settles.

stacking layers

Put a layer of neurons between the input and the output, and XOR becomes easy. The middle (hidden) neurons build their own features, and the output combines them.

The hard question was how to train the hidden layer. When the output is wrong, which hidden weights are to blame, and by how much?

backpropagation

Rumelhart, Hinton and Williams's answer: send the error backwards, layer by layer, with the chain rule. Each layer z = x·W + b, a = f(z) does three things:

  1. turn the error coming from above into the error before the activation: δ = error × f'(z)
  2. work out its own weight gradient: xᵀ · δ
  3. pass the error further down: δ · Wᵀ

One backward pass costs about the same as one forward pass and gives the gradient of every weight. It's the same backward idea as autograd (module 0).

The paper's real point was what this makes possible: the hidden neurons learn useful features on their own that nobody designed. Watch the boundary bend as training goes on:

the network learns the spiral

loss 0.647 · accuracy 0.634

● filled class · ○ hollow class · darker background = more confident it's the filled class

these are snapshots of one training run. at first the boundary is nearly a straight line. as training goes on, the hidden layers learn to bend it, and it wraps around both arms of the spiral.

how the code works

idea where
the perceptron rule Perceptron.fit
weight gradient xᵀδ, error passed down δWᵀ Dense.backward in layers.py
δ = error × f'(z) ActivationLayer.backward
go backwards through every layer Sequential.backward
the error at the output for sigmoid or softmax with cross-entropy it's simply p − y
the training loop Trainer.fit

Each layer saves what its backward pass needs (its input, its value before the activation) during the forward pass. So the whole network is just a list of layers, and the CNN reuses the same Sequential and Trainer.

what the demo checks

Run uv run python examples/11_mlp.py.

  • The perceptron finishes on separable data after 10 updates, well inside the theoretical limit of 356.
  • It never settles on XOR. One hidden layer of 4 neurons solves it exactly.
  • Every layer's backward pass matches a numerical estimate to about 10⁻⁹.
  • The hand-written backpropagation gives exactly the same gradients as the autograd engine from module 0.
  • On a spiral: a straight-line model gets 0.651, one hidden layer of 8 neurons 0.907, of 32 neurons 0.995, and two layers 0.998.

when it goes wrong

  • The perceptron on XOR: it keeps cycling forever.
  • Too few hidden neurons: the boundary can't bend enough.
  • A wrong gradient: the classic bug. Always check against a numerical estimate.