← module 02 · logistic regression
paywalled

The Regression Analysis of Binary Sequences

David R. Cox

Journal of the Royal Statistical Society, Series B 20(2):215-242 (with discussion)

about this write-up

No legal free copy was found. This write-up is from secondary sources and standard accounts of the history of logistic regression, not from a reading of the original PDF.

the original →

the background

Yes/no outcomes were already studied in drug testing: give a dose, count how many respond. The standard tool was the probit model, which uses the bell curve's S-shape. Others had argued for the logistic S-curve and coined the word "logit".

Cox turned this into a general regression method for yes/no data: any number of inputs, and outcomes that may come in a sequence over time.

the model

For each trial, with success probability θ and input x:

log⁡θ1−θ=α+βx\log\frac{\theta}{1-\theta} = \alpha + \beta x

the main argument: the maths gets simple

The logistic and probit curves look almost the same, so data alone can't easily choose between them. Cox's point: with the logistic curve, the data only matters through two simple sums, Σy and Σxy. The log-likelihood is

ℓ(α,β)=α∑yi+β∑xiyi−∑log⁡(1+eα+βxi)\ell(\alpha,\beta) = \alpha\sum y_i + \beta\sum x_i y_i - \sum \log(1 + e^{\alpha+\beta x_i})

That makes fitting straightforward, and it allows exact tests. The probit model has no such shortcut.

"sequences"

The title refers to yes/no outcomes observed in order, where the chance can depend on time or on earlier outcomes (for example, on whether the previous trial succeeded).

why it lasted

  • Each weight is a log odds ratio, which is easy to explain.
  • The loss has one bottom and no traps, so Newton's method finds it reliably.
  • It became the model for yes/no data in the wider family of generalised linear models.
  • In machine learning it's the sigmoid output with cross-entropy loss, and softmax for more than two classes.

in this repo

  • LogisticRegression(solver="newton") is the classical fit, and it reports standard errors (se_W, se_b).
  • The logistic demo shows those standard errors match the real spread of the weights over 300 simulated datasets.