Lecture 6e: rmsprop: Divide the gradient by a running average of its recent magnitude
Geoffrey Hinton, Nitish Srivastava, Kevin Swersky
Coursera: Neural Networks for Machine Learning (lecture slides, never formally published)
about this write-up
No paper exists. The slides are linked above (hosted on a Toronto course page; the original Coursera course is gone). Often cited as 'Tieleman & Hinton'.
the original →no paper exists
RMSProp was introduced on one slide of an online course lecture and spread from there. It's usually cited as "Tieleman & Hinton, lecture 6.5". The slides themselves also credit Srivastava and Swersky.
the problem it addressed
Gradient sizes vary enormously, both between weights and over time. One global step size is too big for some weights and too small for others.
An earlier method, Rprop, used only the sign of each gradient, with a step size per weight that grew while the sign kept agreeing. That works when you use all the data at once, but with mini-batches the sign keeps flipping from batch to batch and Rprop breaks down.
the idea
Keep Rprop's goal of ignoring how big the gradient is, but do it in a mini-batch- friendly way: divide by a running average of recent gradient sizes.
- If a weight's gradients are always large,
sis large and its steps shrink. - If they're tiny, its steps grow.
- Averaging over recent batches means one noisy batch doesn't take over.
It's close to AdaGrad, which divides by the total of all past squared gradients. That total only grows, so AdaGrad's steps shrink toward zero over time. RMSProp's running average forgets the distant past, so the steps stay useful.
legacy
Adam combines RMSProp's per-weight scaling with momentum and a fix for the averages starting at zero. Most adaptive optimisers in use today come from this one slide.
in this repo
RMSProp(lr, rho=0.9)inoptimizers.py.- In the optimisers demo it needs 141 steps to cross the narrow valley (plain gradient descent: 559), and 64 epochs to train the spiral network (plain SGD: 161).