module 02

Logistic Regression

paywalled

The Regression Analysis of Binary Sequences

David R. Cox

How do you model a yes/no outcome with a straight line? Put the line on the log-odds scale, where it can take any value, then squash it back into a probability. This paper set that up as a general regression method and showed why the logistic curve makes the maths clean.

the problem

The answer is yes or no: spam or not, pass or fail. We want the probability of yes. A straight line can't give that directly, because it happily predicts 1.3 or −0.2.

the idea: put the line somewhere it fits

The odds are p / (1 − p), and their logarithm can be any number at all. So put the straight line there, then squash it back:

log⁡p1−p=w⋅x+b⟺p=σ(w⋅x+b)=11+e−(w⋅x+b)\log\frac{p}{1-p} = w\cdot x + b \quad\Longleftrightarrow\quad p = \sigma(w\cdot x + b) = \frac{1}{1 + e^{-(w\cdot x + b)}}

The squashing function σ is the S-shaped sigmoid. Reading a weight is easy: increasing input j by 1 multiplies the odds by e^{w_j}. That's why this model is still everywhere in medicine and the social sciences.

Cox's argument for this particular S-curve, over other similar ones, was that it makes the maths clean: the data only enters through a couple of simple sums.

learning the weights

We pick the weights that make the observed answers most likely. That means minimising the log loss:

L(w)=−1n∑i[yilog⁡pi+(1−yi)log⁡(1−pi)]L(w) = -\frac1n\sum_i \big[y_i\log p_i + (1-y_i)\log(1-p_i)\big]

Its gradient is remarkably simple: prediction minus truth, times the input, Xᵀ(p − y)/n. Watch the boundary settle as the steps go by:

the boundary moves as the model learns

loss 0.581 · accuracy 0.919

● filled class · ○ hollow class · darker background = more confident it's the filled class

after one step the boundary is in roughly the right place but very soft (wide grey band). with more steps it settles where the classes meet and gets sharper, because the model grows more confident. the boundary is always a straight line.

There's also a much faster method, Newton's method: use the curvature of the loss as well as its slope. It reaches the answer in a handful of steps, and as a bonus the curvature tells you how sure you can be about each weight (its standard error).

more than two classes

Give each class its own score and turn the scores into probabilities with softmax (see the activations page). The gradient keeps the same simple form, which is why one line of code handles both cases.

how the code works

idea in logistic.py
sigmoid / softmax _proba
log loss nll
the gradient Xᵀ(p − y)/n the G = … line in fit
Newton's method _newton, which also fills se_W and se_b
the yes/no cut-off threshold (0.5 by default)
ROC curve and AUC roc_curve, auc

what the demo checks

Run uv run python examples/02_logistic.py.

  • Gradient descent needs 3000 steps; Newton's method needs 7 to reach the same answer.
  • The hand-derived gradient matches a numerical estimate to 3 × 10⁻¹¹.
  • The standard errors from one fit (0.158, 0.196) match the real spread across 300 fresh datasets (0.166, 0.204).
  • If the two classes can be split perfectly, the weights grow forever (5.5 → 12.4 → 27.5) because a sharper boundary always looks better. A small penalty holds them at about 3.9.
  • Moving the cut-off from 0.2 to 0.8 trades catching every yes (recall 0.99) for being right when you say yes (precision 0.92).

when it goes wrong

  • Perfectly separable data: the weights never stop growing without a penalty.
  • 0.5 isn't a magic cut-off: with rare classes, a different threshold is often better.
  • The boundary is always a straight line. Circles or XOR need extra features, or a neural network.