Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
John S. Bridle
Neurocomputing: Algorithms, Architectures and Applications (F. Fogelman Soulié & J. Hérault, eds.), NATO ASI Series F vol. 68, Springer, pp. 227-236
about this write-up
Book chapter, paywalled. A related Bridle paper (NIPS 2) is free; it is linked on the activations page. This write-up is from secondary sources, not from a reading of the chapter.
the original →the question
A classification network gives one number per class. What do those numbers mean, and how should the network be trained? Squared error against 0/1 targets worked, but it treated the outputs as unrelated numbers.
the answer
-
Turn the outputs into probabilities. Make each one positive and divide by the total:
All outputs are positive and add up to 1. Bridle named this softmax: a smooth, trainable version of "pick the biggest".
-
Train it as a probability. Make the correct class as likely as possible (minimise cross-entropy) instead of using squared error. The outputs can then be read as the chance of each class given the input, which connects networks to classical statistics.
why it works so well
- With cross-entropy, the gradient through softmax is simply
p − y: no flat spots from the output layer, and a very simple backward pass. - With two classes, softmax becomes the sigmoid, so it's the natural extension of logistic regression.
- A temperature
T(softmax ofz/T) moves the output between almost one-hot (smallT) and almost even (largeT). This is used a lot today, for example when sampling text.
in this repo
activations.softmax(z, temperature)subtracts the largest score first, which changes nothing but stops overflow.losses.cross_entropy_with_logitsreturnsp − yas the gradient.- The activations page has a figure showing the temperature effect.