module 11
Optimizers: Momentum, RMSProp, Adam, AdamW
Some Methods of Speeding up the Convergence of Iteration Methods
Boris T. Polyak
Adds momentum to gradient descent, like a heavy ball rolling downhill. It stops zig- zagging across narrow valleys and speeds up along them.
On the Importance of Initialization and Momentum in Deep Learning
Ilya Sutskever et al.
Shows that plain gradient descent with momentum (especially Nesterov's version) and a careful start can train deep networks people thought needed far more complex methods.
Lecture 6e: rmsprop: Divide the gradient by a running average of its recent magnitude
Geoffrey Hinton et al.
From a lecture slide, not a paper: divide each weight's gradient by the recent average size of that weight's gradients, so every weight effectively gets its own step size.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma & Jimmy Ba
Combines momentum with RMSProp's per-weight step sizes, and corrects for both starting at zero. It works well with little tuning and became the default optimiser.
Decoupled Weight Decay Regularization
Ilya Loshchilov & Frank Hutter
Shows that inside Adam, adding a penalty to the gradient doesn't actually shrink weights evenly. AdamW applies the shrinking directly to the weights instead, and generalises better.
the two problems with plain gradient descent
Plain gradient descent takes a step downhill: w ← w − η·gradient. It has two
weaknesses:
- In a long narrow valley it bounces between the steep walls and crawls along the floor.
- One step size has to serve every weight, even when their gradients differ by a thousand times.
Momentum fixes the first. RMSProp and Adam fix the second. AdamW fixes something Adam broke.
four optimisers in the same narrow valley
momentum: a heavy ball
Keep a running velocity and let each gradient push it:
Across the valley, the gradients keep flipping sign, so they cancel out. Along the floor they all point the same way, so they add up. Polyak showed this turns the slowdown from a narrow valley into roughly its square root: a valley 100 times steeper one way becomes only about 10 times slower to cross, not 100.
Nesterov momentum looks ahead: it takes the gradient at the point where momentum is about to carry you. Sutskever and colleagues showed momentum plus a careful start can train deep networks people thought needed much fancier methods.
rmsprop: a step size for every weight
Divide each weight's gradient by the recent average size of that weight's gradients:
A weight with huge gradients takes small steps; a weight with tiny gradients takes bigger ones. This came from a lecture slide, not a paper.
adam: both at once
Adam keeps a momentum-style average of the gradient and an RMSProp-style average of its size, then divides one by the other.
Both averages start at zero, so early on they're too small. Adam divides that bias out. Without the fix, the very first steps would be about 3 times too big.
A nice side effect: multiply the whole loss by 1000 and Adam takes exactly the same
steps. The step size is about η no matter how big the gradients are.
adamw: shrinking weights properly
A common trick is to keep weights small by shrinking them a little every step (weight decay). With plain gradient descent, adding a penalty to the gradient does the same thing. Inside Adam it doesn't: the penalty gets divided by the same per-weight size, so weights with large or noisy gradients barely shrink.
AdamW applies the shrinking directly to the weights, outside Adam's rescaling, so every weight shrinks at the same rate. It generalises better and became the default for training large models.
how the code works
| idea | in optimizers.py |
|---|---|
| per-weight memory (velocity, averages) | Optimizer._buf |
| momentum and Nesterov | SGD(momentum=…, nesterov=…) |
| per-weight step size | RMSProp |
| both, with the bias fix | Adam.direction |
| penalty inside the gradient | Adam(weight_decay=…) |
| shrinking applied directly | AdamW |
what the demo checks
Run uv run python examples/11_optimizers.py.
- In a valley 100 times steeper one way, steps to reach the bottom: plain gradient descent 559, momentum 81, Nesterov 98, RMSProp 141, Adam 172.
- Momentum's measured speed (0.8210 per step) matches Polyak's prediction (0.8182).
- Multiply the loss by 1000: plain gradient descent blows up, Adam doesn't change at all.
- A weight that should shrink from 5 toward 0 under noisy gradients: AdamW gets it to 0.66 regardless of the noise. Adam with the penalty in the gradient doesn't shrink it at all when the noise is large.
- Training a spiral network, epochs to get the loss under 0.05: plain SGD 161, momentum 23, RMSProp 64, Adam 48.
when it goes wrong
- Narrow valleys with plain gradient descent: bouncing.
- A learning rate too big for plain gradient descent: blow-up. Adam is much more forgiving.
- Weight decay inside Adam: use AdamW instead.
- Momentum too high: overshooting and circling around the bottom.