module 11

Activation Functions

free pdf

Efficient BackProp

Yann LeCun et al.

A practical guide to training networks well: centre and scale your inputs, prefer tanh over sigmoid, and start the weights at a sensible size. It explains why sigmoid layers make gradients shrink.

paywalled

Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition

John S. Bridle

Introduces softmax, which turns a network's raw scores into probabilities that add up to one, and argues outputs should be read as class probabilities and trained that way.

free pdf

Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters

John S. Bridle

A companion paper from the same year that uses the softmax output to train speech- recognition models as networks, so the model learns to tell the right answer apart from the wrong ones.

free pdf

Rectified Linear Units Improve Restricted Boltzmann Machines

Vinod Nair & Geoffrey E. Hinton

Introduces ReLU, max(0, x): pass positive values through and zero out negatives. It is simple, fast, and doesn't flatten out for large inputs the way sigmoid does.

free pdf

Deep Sparse Rectifier Neural Networks

Xavier Glorot et al.

Shows ReLU lets deep networks train well without special tricks, and that many units output exactly zero, which makes the network sparse and efficient.

free pdf

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Andrew L. Maas et al.

Introduces leaky ReLU: give negative inputs a small slope instead of zero, so a unit that goes negative can still learn and never dies completely.

free pdf

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He et al.

Introduces PReLU, where each unit learns its own slope for negative inputs instead of using a fixed one.

free pdf

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Djork-Arné Clevert et al.

Introduces ELU: like ReLU for positive inputs, but a smooth curve below zero. The negative outputs keep each layer's average near zero, which speeds up learning.

free pdf

Gaussian Error Linear Units (GELUs)

Dan Hendrycks & Kevin Gimpel

Introduces GELU, a smooth version of ReLU that scales each input by how likely it is to be positive. It is now the standard choice in transformers.

free pdf

Searching for Activation Functions

Prajit Ramachandran et al.

Used an automated search to try many formulas for activation functions. The winner, Swish (x times sigmoid of x), beat ReLU on several tasks.

why a network needs them

A layer multiplies its input by weights and adds a bias. Stack ten of those and you still only have one big multiply-and-add, a straight line. The activation function is the bend applied after each layer. Without it, depth adds nothing.

But the bend has a cost. When the network learns, the error signal travels backwards through every layer, and at each one it gets multiplied by the activation's slope. If the slope is small, the signal shrinks a little at every layer until nothing reaches the early layers. Most of the history below is about keeping that slope healthy.

Pick a function below. The solid line is the function; the dashed line is its slope.

activation functions and their slopes

-4-2024-10123f(z)slope
sigmoid(z) = 1 / (1 + e^-z)
flat at both ends, so the slope (dashed) is almost zero there. it never goes above 0.25, which is why stacking many sigmoid layers makes gradients vanish. max slope: 0.250

the s-curves: sigmoid and tanh

These were the original choices. Both squash any input into a small range, and both go flat at the ends, where the slope is almost zero.

  • sigmoid outputs between 0 and 1. Its slope is never more than 0.25, so ten sigmoid layers can shrink the error signal by about a million times. That's the vanishing gradient.
  • tanh outputs between −1 and 1 and is centred on zero, which LeCun and colleagues recommended in Efficient BackProp. Its slope reaches 1 in the middle, but it still goes flat at the ends.

relu and its family

ReLU is just max(0, z): keep positive numbers, zero out negative ones.

  • For positive inputs the slope is exactly 1, so the error signal passes through untouched. Deep networks suddenly became easy to train.
  • Many units output exactly 0, which makes the network sparse and cheap.
  • The catch: a unit whose input is always negative has slope 0 forever. It never learns again. That's a dead unit.

The fixes all give negative inputs a little life:

function what it does below zero
leaky relu a small fixed slope, like 0.01
prelu a slope the network learns for each unit
elu a smooth curve that settles at −1

smooth versions: gelu and swish

GELU keeps each input in proportion to how likely it is to be positive, a soft version of ReLU's hard on/off switch. Swish (z · sigmoid(z)) was found by letting a computer search through many formulas. Both are smooth, both dip slightly below zero before rising, and GELU is now the default in transformers.

softmax: the output layer

For classification, the last layer gives one score per class. Softmax turns those scores into probabilities: make each score positive with e^score, then divide by the total so they add up to 1. A temperature controls how sharp the result is.

softmax turns scores into probabilities

0.66
score 2
0.24
score 1
0.10
score 0.1
the same three scores (2, 1, 0.1). a low temperature makes the biggest score take almost everything; a high temperature spreads the probability out. the bars always add up to 1.

With cross-entropy loss, the gradient through softmax works out to simply prediction minus truth (p − y), which is why the two are always used together.

how the code works

Every activation in activations.py has two methods: forward(z) gives the value and derivative(z) gives the slope. Both take the input z directly, so nothing is hidden or cached. PReLU is the one exception: it stores its learned slope a, which the training loop updates like any other weight.

softmax subtracts the largest score before taking e^score. That changes nothing in the result but stops huge numbers from overflowing: softmax([1000, 999]) works.

what the demo checks

Run uv run python examples/11_activations.py.

  • Every slope is correct: each derivative matches a numerical estimate.
  • Vanishing gradients, measured: in a 10-layer sigmoid network, the error signal reaching the first layer is ten million times smaller than at the last layer. With ReLU it's only about 37 times smaller.
  • Dead units: push every ReLU unit into its negative zone and none of them ever turn on again, so accuracy stays at a coin flip (0.51). Leaky ReLU and ELU recover to 1.0.
  • Fair comparison on a spiral, 5 seeds each: the ReLU family, GELU and Swish solve it 5 out of 5 times. Sigmoid solves it 0 times, tanh once.
  • But tanh isn't really the problem. With only two inputs, the usual "Xavier" starting weights are tiny, so every tanh stays in its straight middle part and the network behaves like a straight line. With bigger starting weights tanh solves it 5 out of 5.

when it goes wrong

  • Deep sigmoid networks: the early layers barely learn.
  • Dead ReLUs: units that never turn back on. Use leaky ReLU or ELU.
  • Starting weights that don't fit the activation: tanh with too-small weights stays stuck in its linear zone.
  • Softmax without subtracting the max: overflow on large scores.