module 11

Training Techniques: Init, Dropout, Batch Norm, Schedules, Early Stopping

free pdf

Understanding the Difficulty of Training Deep Feedforward Neural Networks

Xavier Glorot & Yoshua Bengio

Studies why deep networks were hard to train and proposes starting weights at a size that keeps signals from fading or blowing up as they pass through the layers (Xavier initialisation).

free pdf

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He et al.

Adjusts the starting weight size for ReLU, which zeroes half its inputs, by doubling the variance (He initialisation). This let very deep ReLU networks train from scratch.

free pdf

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava et al.

While training, randomly switch off units. The network can't rely on any one unit, so it learns sturdier features and overfits less.

free pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe & Christian Szegedy

Rescales each layer's outputs to a standard range for every mini-batch. Training becomes much faster and less sensitive to the learning rate.

free pdf

Cyclical Learning Rates for Training Neural Networks

Leslie N. Smith

Instead of hunting for one perfect learning rate, sweep it up and down between two values. It often trains as well with far less tuning.

free pdf

SGDR: Stochastic Gradient Descent with Warm Restarts

Ilya Loshchilov & Frank Hutter

Lower the learning rate along a smooth cosine curve, then jump it back up and repeat. The restarts help escape poor solutions.

free pdf

Early Stopping - But When?

Lutz Prechelt

Watch the error on held-out data and stop training once it starts rising. The paper compares concrete rules for deciding exactly when.

starting weights (initialisation)

At every layer the signal is multiplied by random weights. If that multiplication shrinks the signal a little, after 20 layers there's nothing left. If it grows the signal, it explodes. The fix is to choose how big the starting weights are, so the signal keeps its size on the way forward and on the way back.

  • Xavier (Glorot and Bengio): weight variance 2 / (inputs + outputs). Made for activations like tanh.
  • He: ReLU throws away half its input, which halves the signal at every layer, so double the variance: 2 / inputs. This let very deep ReLU networks train from scratch.

how big the signal is at each of 20 layers

510152010^-2010^-1510^-1010^-510^0layersignal size (log scale)
tanh + tiny randomtanh + xavierrelu + xavierrelu + he
with tiny random weights the signal shrinks to almost nothing within a few layers. xavier keeps tanh alive. relu throws away half its input, so with xavier the signal halves at every layer; he's version doubles the weights to make up for it and keeps the signal steady.

dropout

While training, switch off each neuron at random with probability p. Every batch then trains a slightly different, thinned-out network, and no neuron can rely on a particular partner being there. The network learns sturdier features and overfits less.

At test time you use the whole network. The version here scales the surviving neurons up by 1/(1−p) during training, so nothing needs to change at test time.

batch normalisation

For every feature, rescale its values across the current batch to an average of 0 and a spread of 1, then let the network apply its own scale and shift:

x^=x−batch meanbatch spread,y=γx^+β\hat x = \frac{x - \text{batch mean}}{\text{batch spread}},\qquad y = \gamma\hat x + \beta

At test time there's no batch, so it uses running averages collected during training. The original explanation for why it works is still debated, but the effect isn't: training gets much faster and tolerates bigger learning rates.

learning-rate schedules

Instead of one fixed learning rate, change it as training goes:

  • cyclical (Smith): sweep it up and down between two values. It often works as well as a carefully tuned fixed rate, with much less tuning.
  • cosine with restarts (Loshchilov and Hutter): glide down a smooth curve, then jump back to the top. The restarts help escape poor solutions.

two learning-rate schedules

010203040506000.0050.01training steplearning rate
cyclicalcosine with restarts
cyclical: the learning rate climbs and falls in a steady triangle. cosine with restarts: it glides down a smooth curve, then jumps back to the top, and each round is twice as long as the last.

early stopping

Error on held-out data usually falls, then starts rising as the network begins to memorise noise. Stop near the turn. Prechelt compared concrete stopping rules, such as "stop when the held-out error is more than 5% above the best so far", and found that waiting longer buys a little accuracy for a lot of extra training time.

how the code works

idea where
Xavier, He and LeCun starting weights init.py
dropout (does nothing at test time) layers.Dropout
batch normalisation with running averages layers.BatchNorm
cyclical and cosine schedules schedules.cyclical, schedules.cosine_restarts
early stopping (patience or Prechelt's rule) schedules.EarlyStopping
all wired into training MLP(init=…, dropout=…, batch_norm=…, lr_schedule=…, early_stopping=…)

The hard part of batch norm is its backward pass: the batch average depends on every example, so each example's gradient involves all the others. The paper's three equations collapse into one line in the code.

what the demo checks

Run uv run python examples/11_training.py.

  • Starting weights over 20 layers: tiny random weights shrink the signal to 10⁻²⁰. Xavier keeps tanh alive. ReLU with Xavier halves every layer. ReLU with He stays near 1 all the way down.
  • Dropout keeps the average: the mean over 20,000 training passes matches the test-time output.
  • Dropout on noisy data (80 points, 15% wrong labels): it gives up some training accuracy and gains test accuracy (best run: 0.870 → 0.907). The best p changed from one data draw to the next.
  • Batch norm in a 6-layer sigmoid network: with it, every run reaches a low loss within 10 epochs; without it, some runs are still stuck after 60.
  • Early stopping: held-out error is lowest at epoch 4, then more than doubles by epoch 400. Accuracy barely differs, but the long-trained model is badly overconfident.

A note on how these demos were built. The first version used an 80-point spiral, which is too hard to learn from so few points. Dropout seemed to hurt and early stopping fired at epoch 1. The test wasn't measuring the technique, so I switched to noisy moons, where overfitting is the real risk.

when it goes wrong

  • Starting weights that don't fit the activation: the signal fades or explodes.
  • Dropout too high: the network can't even fit the clean signal.
  • Batch norm with tiny batches: the batch averages get noisy.
  • Forgetting to switch to test mode: dropout and batch norm behave differently then, and leaving training mode on is a classic bug.
  • Early stopping keeps the last weights here, not the best ones. Saving the best version is the usual improvement.