module 12
Convolutional Neural Networks
Gradient-Based Learning Applied to Document Recognition
Yann LeCun et al.
Introduces LeNet: small filters that slide across an image and reuse the same weights everywhere, with pooling in between, all trained end to end. It read handwritten digits and cheques, and introduced the MNIST dataset.
the problem with ordinary layers on images
Feed an image to an ordinary (fully connected) layer and every neuron gets its own weight for every pixel:
- Far too many weights. One such layer producing 24×24×8 outputs from a 28×28 image needs about 3.6 million weights.
- No idea what "nearby" means. Shuffle the pixels the same way in every image and it learns just as well. It ignores the picture entirely.
- No reuse. A stroke detector learned in the top-left must be learned again, from scratch, for the bottom-right.
LeNet's three ideas
- Look at small patches. Each neuron sees a small window, say 3×3 or 5×5, because edges, corners and stroke ends are local.
- Share the weights. Every neuron in a layer uses the same small filter, slid across the whole image. That's a convolution. One 5×5 filter is 26 numbers, wherever it's used. Shift the input and the output shifts with it.
- Shrink as you go (pooling). Reduce each map's size, so the exact position of a feature matters less and later layers see a wider area.
Stack them (convolve, pool, convolve, pool, then ordinary layers) and train everything together with backpropagation. The filters are learned, not designed.
what the first-layer filters see
input
8 learned filters (top) and where each one lights up (bottom)
convolution as one big multiply
The obvious way to convolve is four nested loops. The fast way copies every small window into one row of a big matrix (im2col). The whole convolution is then one matrix multiply.
Going backward is the reverse: spread the gradient rows back onto the image, and add them up where windows overlap. An input pixel used by several windows gets gradient from all of them, which is exactly what the chain rule asks for.
The output size, for input size H, filter size k, step s and padding p:
how the code works
| idea | where |
|---|---|
| convolution, forward and back | layers.Conv2D (im2col, multiply, col2im) |
| max pooling (gradient goes only to the biggest value) | layers.MaxPool2D |
| average pooling (gradient shared equally) | layers.AvgPool2D |
| global max pooling (a later idea, not in LeNet) | layers.GlobalMaxPool, CNN(head="global_max") |
| the network | cnn.CNN |
| training | reuses Trainer and Sequential from the MLP |
| looking inside | feature_maps(X), learned_filters() |
Simpler than the original LeNet: ReLU instead of a scaled tanh, plain max or average pooling instead of trainable pooling, every map connected to every map, and softmax at the output. The core ideas are the same.
what the demo checks
Run uv run python examples/12_cnn.py (about a minute). There's no dataset: the
images are synthetic 12×12 pictures of a horizontal bar, a vertical bar or a plus,
with noise, at random places.
-
The one-big-multiply convolution matches the four nested loops exactly.
-
Every backward pass matches a numerical estimate to about 10⁻⁹.
-
Weight sharing in numbers: 208 weights for a 5×5 convolution with 8 filters, against 3,617,280 for an ordinary layer with the same output.
-
Train on the left half of the image, test on the right half:
model left (seen places) right (new places) CNN with LeNet's ordinary final layers 0.997 0.308 CNN with global pooling at the end 0.998 0.995 CNN, LeNet ending, trained on all places 0.993 0.990 ordinary network on raw pixels 0.787 0.332 I expected the plain CNN to handle new places, and it didn't. The convolution layers don't care where a bar is, but LeNet's final ordinary layer has one weight per position, and the right-half positions never lit up in training. Global pooling (keep only each filter's strongest response, wherever it was) fixes that, with three times fewer weights. So does training on all positions.
when it goes wrong
- Patterns in new places: see the table above.
- Steps too big or no padding: small features at the edges get skipped.
- Too few filters: not enough different features to tell the classes apart.