← module 01 · linear regression, ridge and lasso
paywalled

Ridge Regression: Biased Estimation for Nonorthogonal Problems

Arthur E. Hoerl, Robert W. Kennard

Technometrics 12(1):55-67

about this write-up

No legal free copy was found. This write-up is from secondary sources and standard textbook treatments (ESL ch. 3), not from a reading of the original PDF.

the original →

the problem they faced

They worked with industrial data where the inputs were strongly related to each other. In that situation, ordinary least squares:

  • is right on average, but swings wildly from one dataset to the next;
  • tends to give weights that are too big: tiny patterns in the data get blown up into huge weights;
  • often gives weights with the wrong sign or a silly size.

the fix

Add a small number k to the diagonal before solving:

β^(k)=(X⊤X+kI)−1X⊤y,k≥0\hat\beta(k) = (X^\top X + kI)^{-1} X^\top y, \qquad k \ge 0

k = 0 is ordinary least squares. As k grows, the weights shrink toward zero and stop swinging. The same answer comes from adding a penalty k‖β‖² to the squared error.

the main result

There is always some k > 0 that gives a smaller total error than ordinary least squares. Starting from k = 0, a little penalty adds only a tiny bit of bias but removes a lot of swing. So a small amount of shrinking always pays off.

The proof says such a k exists, not which one, because that depends on the true answer, which you don't know.

choosing k: the ridge trace

Plot every weight against k. For small k the weights jump around; past some point they settle down. Pick k where they've settled, the signs make sense and the error hasn't grown much. Today people usually choose k with cross-validation instead, but the ridge trace is still the best picture of what ridge does.

in this repo

  • Ridge(alpha) solves (XᵀX + n·α·I) w = Xᵀy.
  • ridge_trace(X, y, alphas) draws the weight-vs-penalty curves; the linear page shows them in a figure.
  • The linear demo reproduces the paper's main point: on two near-copy inputs, plain weights swing by about ±5.7 between resamples, and ridge's by about ±0.05.