module 00

Automatic Differentiation

free pdf

Automatic Differentiation in Machine Learning: a Survey

Atılım Güneş Baydin et al.

Explains how a computer can get exact derivatives of any program, not by doing calculus on a formula but by recording each small step the program takes and applying the chain rule backwards through them. It shows that backpropagation is just one case of this idea, and why going backwards is so cheap when you have millions of weights and one loss.

why this comes first

Every network on this site learns the same way: work out how much each weight contributed to the error, then nudge it. That "how much" is a derivative, and with thousands of weights you need all of them, fast and exact. Automatic differentiation (autograd) is how the computer gets them.

three ways to get a derivative

way how problem
numerical nudge the input a tiny bit and see how much the output moves slightly wrong, and you need one try per weight
symbolic rewrite the formula with calculus rules, like a maths app formulas balloon, and loops and ifs are awkward
automatic record every small step the program actually ran, then apply the chain rule exact and cheap, at the cost of some memory

The trick: autograd doesn't differentiate a formula, it differentiates what the program did. Any code made of simple operations (+, ×, exp, …) can be differentiated, loops and branches included.

the graph

Running y = tanh(x1·x2 + x1²) creates a small graph: each box is one operation, and the arrows show what feeds into what.

the computation graph for y = tanh(x1·x2 + x1²)

x12.000x2-3.000*-6.000^24.000+-2.000tanh-0.964
going forward, each box works out its value from the boxes feeding into it. x1 = 2 and x2 = −3 give y ≈ −0.964.

Forward, every box computes its value from its inputs.

Backward, starting from y, each box passes a gradient back to its inputs using its own local rule (for a × b, the gradient for a is b, and the other way round). When a value feeds into two places, like x1 here, its gradient is the sum of what comes back from both.

why backward and not forward

You can also push derivatives forward through the graph, but each forward pass gives the derivative with respect to just one input. A backward pass gives the derivative of one output with respect to every input at once.

A network has millions of inputs (its weights) and one output (the loss). So one backward pass does the work of millions of forward ones. That's why training neural networks is possible at all. Backpropagation is exactly this backward pass.

The price is memory: every value from the forward pass has to be kept until the backward pass needs it.

how the code works

idea in autograd.py
a box in the graph Value (one number) or Tensor (an array); _prev holds its inputs
each operation's local rule the small _backward function each operation attaches
the gradient .grad
"sum when used twice" self.grad += … (+=, never =)
visit boxes in the right order _toposort, then walk it backwards in backward()
the slow check numerical_grad

Three details that trip people up:

  • Use +=. With =, a value used twice keeps only one of its gradients. This is the most common autograd bug.
  • Reset first. backward() sets every .grad to zero before starting, or a second call would add on top of the first.
  • Broadcasting becomes summing. Adding a bias of shape (3,) to a batch of shape (N, 3) copies it N times going forward, so its gradient is a sum over the batch going backward (_sum_to).

what the demo checks

Run uv run python examples/00_autograd.py.

  • The gradients for x1 and x2 match a by-hand calculation: 0.070651 and 0.141302.
  • The array version matches the slow numerical estimate to 5 × 10⁻¹¹.

limits

  • Value handles one number at a time, so it's far too slow for real networks. The network modules use hand-written backward passes and use this engine only to check them.
  • It doesn't do second derivatives.