module 11
Training Techniques: Init, Dropout, Batch Norm, Schedules, Early Stopping
Understanding the Difficulty of Training Deep Feedforward Neural Networks
Xavier Glorot & Yoshua Bengio
Studies why deep networks were hard to train and proposes starting weights at a size that keeps signals from fading or blowing up as they pass through the layers (Xavier initialisation).
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He et al.
Adjusts the starting weight size for ReLU, which zeroes half its inputs, by doubling the variance (He initialisation). This let very deep ReLU networks train from scratch.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava et al.
While training, randomly switch off units. The network can't rely on any one unit, so it learns sturdier features and overfits less.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe & Christian Szegedy
Rescales each layer's outputs to a standard range for every mini-batch. Training becomes much faster and less sensitive to the learning rate.
Cyclical Learning Rates for Training Neural Networks
Leslie N. Smith
Instead of hunting for one perfect learning rate, sweep it up and down between two values. It often trains as well with far less tuning.
SGDR: Stochastic Gradient Descent with Warm Restarts
Ilya Loshchilov & Frank Hutter
Lower the learning rate along a smooth cosine curve, then jump it back up and repeat. The restarts help escape poor solutions.
Early Stopping - But When?
Lutz Prechelt
Watch the error on held-out data and stop training once it starts rising. The paper compares concrete rules for deciding exactly when.
starting weights (initialisation)
At every layer the signal is multiplied by random weights. If that multiplication shrinks the signal a little, after 20 layers there's nothing left. If it grows the signal, it explodes. The fix is to choose how big the starting weights are, so the signal keeps its size on the way forward and on the way back.
- Xavier (Glorot and Bengio): weight variance
2 / (inputs + outputs). Made for activations like tanh. - He: ReLU throws away half its input, which halves the signal at every layer, so
double the variance:
2 / inputs. This let very deep ReLU networks train from scratch.
how big the signal is at each of 20 layers
dropout
While training, switch off each neuron at random with probability p. Every batch
then trains a slightly different, thinned-out network, and no neuron can rely on a
particular partner being there. The network learns sturdier features and overfits
less.
At test time you use the whole network. The version here scales the surviving neurons
up by 1/(1−p) during training, so nothing needs to change at test time.
batch normalisation
For every feature, rescale its values across the current batch to an average of 0 and a spread of 1, then let the network apply its own scale and shift:
At test time there's no batch, so it uses running averages collected during training. The original explanation for why it works is still debated, but the effect isn't: training gets much faster and tolerates bigger learning rates.
learning-rate schedules
Instead of one fixed learning rate, change it as training goes:
- cyclical (Smith): sweep it up and down between two values. It often works as well as a carefully tuned fixed rate, with much less tuning.
- cosine with restarts (Loshchilov and Hutter): glide down a smooth curve, then jump back to the top. The restarts help escape poor solutions.
two learning-rate schedules
early stopping
Error on held-out data usually falls, then starts rising as the network begins to memorise noise. Stop near the turn. Prechelt compared concrete stopping rules, such as "stop when the held-out error is more than 5% above the best so far", and found that waiting longer buys a little accuracy for a lot of extra training time.
how the code works
| idea | where |
|---|---|
| Xavier, He and LeCun starting weights | init.py |
| dropout (does nothing at test time) | layers.Dropout |
| batch normalisation with running averages | layers.BatchNorm |
| cyclical and cosine schedules | schedules.cyclical, schedules.cosine_restarts |
| early stopping (patience or Prechelt's rule) | schedules.EarlyStopping |
| all wired into training | MLP(init=…, dropout=…, batch_norm=…, lr_schedule=…, early_stopping=…) |
The hard part of batch norm is its backward pass: the batch average depends on every example, so each example's gradient involves all the others. The paper's three equations collapse into one line in the code.
what the demo checks
Run uv run python examples/11_training.py.
- Starting weights over 20 layers: tiny random weights shrink the signal to 10⁻²⁰. Xavier keeps tanh alive. ReLU with Xavier halves every layer. ReLU with He stays near 1 all the way down.
- Dropout keeps the average: the mean over 20,000 training passes matches the test-time output.
- Dropout on noisy data (80 points, 15% wrong labels): it gives up some training
accuracy and gains test accuracy (best run: 0.870 → 0.907). The best
pchanged from one data draw to the next. - Batch norm in a 6-layer sigmoid network: with it, every run reaches a low loss within 10 epochs; without it, some runs are still stuck after 60.
- Early stopping: held-out error is lowest at epoch 4, then more than doubles by epoch 400. Accuracy barely differs, but the long-trained model is badly overconfident.
A note on how these demos were built. The first version used an 80-point spiral, which is too hard to learn from so few points. Dropout seemed to hurt and early stopping fired at epoch 1. The test wasn't measuring the technique, so I switched to noisy moons, where overfitting is the real risk.
when it goes wrong
- Starting weights that don't fit the activation: the signal fades or explodes.
- Dropout too high: the network can't even fit the clean signal.
- Batch norm with tiny batches: the batch averages get noisy.
- Forgetting to switch to test mode: dropout and batch norm behave differently then, and leaving training mode on is a classic bug.
- Early stopping keeps the last weights here, not the best ones. Saving the best version is the usual improvement.