module 00
Automatic Differentiation
Automatic Differentiation in Machine Learning: a Survey
Atılım Güneş Baydin et al.
Explains how a computer can get exact derivatives of any program, not by doing calculus on a formula but by recording each small step the program takes and applying the chain rule backwards through them. It shows that backpropagation is just one case of this idea, and why going backwards is so cheap when you have millions of weights and one loss.
why this comes first
Every network on this site learns the same way: work out how much each weight contributed to the error, then nudge it. That "how much" is a derivative, and with thousands of weights you need all of them, fast and exact. Automatic differentiation (autograd) is how the computer gets them.
three ways to get a derivative
| way | how | problem |
|---|---|---|
| numerical | nudge the input a tiny bit and see how much the output moves | slightly wrong, and you need one try per weight |
| symbolic | rewrite the formula with calculus rules, like a maths app | formulas balloon, and loops and ifs are awkward |
| automatic | record every small step the program actually ran, then apply the chain rule | exact and cheap, at the cost of some memory |
The trick: autograd doesn't differentiate a formula, it differentiates what the
program did. Any code made of simple operations (+, ×, exp, …) can be
differentiated, loops and branches included.
the graph
Running y = tanh(x1·x2 + x1²) creates a small graph: each box is one operation,
and the arrows show what feeds into what.
the computation graph for y = tanh(x1·x2 + x1²)
Forward, every box computes its value from its inputs.
Backward, starting from y, each box passes a gradient back to its inputs using
its own local rule (for a × b, the gradient for a is b, and the other way
round). When a value feeds into two places, like x1 here, its gradient is the
sum of what comes back from both.
why backward and not forward
You can also push derivatives forward through the graph, but each forward pass gives the derivative with respect to just one input. A backward pass gives the derivative of one output with respect to every input at once.
A network has millions of inputs (its weights) and one output (the loss). So one backward pass does the work of millions of forward ones. That's why training neural networks is possible at all. Backpropagation is exactly this backward pass.
The price is memory: every value from the forward pass has to be kept until the backward pass needs it.
how the code works
| idea | in autograd.py |
|---|---|
| a box in the graph | Value (one number) or Tensor (an array); _prev holds its inputs |
| each operation's local rule | the small _backward function each operation attaches |
| the gradient | .grad |
| "sum when used twice" | self.grad += … (+=, never =) |
| visit boxes in the right order | _toposort, then walk it backwards in backward() |
| the slow check | numerical_grad |
Three details that trip people up:
- Use
+=. With=, a value used twice keeps only one of its gradients. This is the most common autograd bug. - Reset first.
backward()sets every.gradto zero before starting, or a second call would add on top of the first. - Broadcasting becomes summing. Adding a bias of shape
(3,)to a batch of shape(N, 3)copies it N times going forward, so its gradient is a sum over the batch going backward (_sum_to).
what the demo checks
Run uv run python examples/00_autograd.py.
- The gradients for
x1andx2match a by-hand calculation: 0.070651 and 0.141302. - The array version matches the slow numerical estimate to 5 × 10⁻¹¹.
limits
Valuehandles one number at a time, so it's far too slow for real networks. The network modules use hand-written backward passes and use this engine only to check them.- It doesn't do second derivatives.