module 11
Activation Functions
Efficient BackProp
Yann LeCun et al.
A practical guide to training networks well: centre and scale your inputs, prefer tanh over sigmoid, and start the weights at a sensible size. It explains why sigmoid layers make gradients shrink.
Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
John S. Bridle
Introduces softmax, which turns a network's raw scores into probabilities that add up to one, and argues outputs should be read as class probabilities and trained that way.
Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters
John S. Bridle
A companion paper from the same year that uses the softmax output to train speech- recognition models as networks, so the model learns to tell the right answer apart from the wrong ones.
Rectified Linear Units Improve Restricted Boltzmann Machines
Vinod Nair & Geoffrey E. Hinton
Introduces ReLU, max(0, x): pass positive values through and zero out negatives. It is simple, fast, and doesn't flatten out for large inputs the way sigmoid does.
Deep Sparse Rectifier Neural Networks
Xavier Glorot et al.
Shows ReLU lets deep networks train well without special tricks, and that many units output exactly zero, which makes the network sparse and efficient.
Rectifier Nonlinearities Improve Neural Network Acoustic Models
Andrew L. Maas et al.
Introduces leaky ReLU: give negative inputs a small slope instead of zero, so a unit that goes negative can still learn and never dies completely.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He et al.
Introduces PReLU, where each unit learns its own slope for negative inputs instead of using a fixed one.
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Djork-Arné Clevert et al.
Introduces ELU: like ReLU for positive inputs, but a smooth curve below zero. The negative outputs keep each layer's average near zero, which speeds up learning.
Gaussian Error Linear Units (GELUs)
Dan Hendrycks & Kevin Gimpel
Introduces GELU, a smooth version of ReLU that scales each input by how likely it is to be positive. It is now the standard choice in transformers.
Searching for Activation Functions
Prajit Ramachandran et al.
Used an automated search to try many formulas for activation functions. The winner, Swish (x times sigmoid of x), beat ReLU on several tasks.
why a network needs them
A layer multiplies its input by weights and adds a bias. Stack ten of those and you still only have one big multiply-and-add, a straight line. The activation function is the bend applied after each layer. Without it, depth adds nothing.
But the bend has a cost. When the network learns, the error signal travels backwards through every layer, and at each one it gets multiplied by the activation's slope. If the slope is small, the signal shrinks a little at every layer until nothing reaches the early layers. Most of the history below is about keeping that slope healthy.
Pick a function below. The solid line is the function; the dashed line is its slope.
activation functions and their slopes
flat at both ends, so the slope (dashed) is almost zero there. it never goes above 0.25, which is why stacking many sigmoid layers makes gradients vanish. max slope: 0.250
the s-curves: sigmoid and tanh
These were the original choices. Both squash any input into a small range, and both go flat at the ends, where the slope is almost zero.
- sigmoid outputs between 0 and 1. Its slope is never more than 0.25, so ten sigmoid layers can shrink the error signal by about a million times. That's the vanishing gradient.
- tanh outputs between −1 and 1 and is centred on zero, which LeCun and colleagues recommended in Efficient BackProp. Its slope reaches 1 in the middle, but it still goes flat at the ends.
relu and its family
ReLU is just max(0, z): keep positive numbers, zero out negative ones.
- For positive inputs the slope is exactly 1, so the error signal passes through untouched. Deep networks suddenly became easy to train.
- Many units output exactly 0, which makes the network sparse and cheap.
- The catch: a unit whose input is always negative has slope 0 forever. It never learns again. That's a dead unit.
The fixes all give negative inputs a little life:
| function | what it does below zero |
|---|---|
| leaky relu | a small fixed slope, like 0.01 |
| prelu | a slope the network learns for each unit |
| elu | a smooth curve that settles at −1 |
smooth versions: gelu and swish
GELU keeps each input in proportion to how likely it is to be positive, a soft
version of ReLU's hard on/off switch. Swish (z · sigmoid(z)) was found by
letting a computer search through many formulas. Both are smooth, both dip slightly
below zero before rising, and GELU is now the default in transformers.
softmax: the output layer
For classification, the last layer gives one score per class. Softmax turns those
scores into probabilities: make each score positive with e^score, then divide by
the total so they add up to 1. A temperature controls how sharp the result is.
softmax turns scores into probabilities
With cross-entropy loss, the gradient through softmax works out to simply
prediction minus truth (p − y), which is why the two are always used together.
how the code works
Every activation in activations.py has two methods: forward(z) gives the value
and derivative(z) gives the slope. Both take the input z directly, so nothing is
hidden or cached. PReLU is the one exception: it stores its learned slope a, which
the training loop updates like any other weight.
softmax subtracts the largest score before taking e^score. That changes nothing
in the result but stops huge numbers from overflowing: softmax([1000, 999]) works.
what the demo checks
Run uv run python examples/11_activations.py.
- Every slope is correct: each
derivativematches a numerical estimate. - Vanishing gradients, measured: in a 10-layer sigmoid network, the error signal reaching the first layer is ten million times smaller than at the last layer. With ReLU it's only about 37 times smaller.
- Dead units: push every ReLU unit into its negative zone and none of them ever turn on again, so accuracy stays at a coin flip (0.51). Leaky ReLU and ELU recover to 1.0.
- Fair comparison on a spiral, 5 seeds each: the ReLU family, GELU and Swish solve it 5 out of 5 times. Sigmoid solves it 0 times, tanh once.
- But tanh isn't really the problem. With only two inputs, the usual "Xavier" starting weights are tiny, so every tanh stays in its straight middle part and the network behaves like a straight line. With bigger starting weights tanh solves it 5 out of 5.
when it goes wrong
- Deep sigmoid networks: the early layers barely learn.
- Dead ReLUs: units that never turn back on. Use leaky ReLU or ELU.
- Starting weights that don't fit the activation: tanh with too-small weights stays stuck in its linear zone.
- Softmax without subtracting the max: overflow on large scores.