Ridge Regression: Biased Estimation for Nonorthogonal Problems
Arthur E. Hoerl, Robert W. Kennard
Technometrics 12(1):55-67
about this write-up
No legal free copy was found. This write-up is from secondary sources and standard textbook treatments (ESL ch. 3), not from a reading of the original PDF.
the original →the problem they faced
They worked with industrial data where the inputs were strongly related to each other. In that situation, ordinary least squares:
- is right on average, but swings wildly from one dataset to the next;
- tends to give weights that are too big: tiny patterns in the data get blown up into huge weights;
- often gives weights with the wrong sign or a silly size.
the fix
Add a small number k to the diagonal before solving:
k = 0 is ordinary least squares. As k grows, the weights shrink toward zero and
stop swinging. The same answer comes from adding a penalty k‖β‖² to the squared
error.
the main result
There is always some k > 0 that gives a smaller total error than ordinary least
squares. Starting from k = 0, a little penalty adds only a tiny bit of bias but
removes a lot of swing. So a small amount of shrinking always pays off.
The proof says such a k exists, not which one, because that depends on the true
answer, which you don't know.
choosing k: the ridge trace
Plot every weight against k. For small k the weights jump around; past some point
they settle down. Pick k where they've settled, the signs make sense and the error
hasn't grown much. Today people usually choose k with cross-validation instead, but
the ridge trace is still the best picture of what ridge does.
in this repo
Ridge(alpha)solves(XᵀX + n·α·I) w = Xᵀy.ridge_trace(X, y, alphas)draws the weight-vs-penalty curves; the linear page shows them in a figure.- The linear demo reproduces the paper's main point: on two near-copy inputs, plain weights swing by about ±5.7 between resamples, and ridge's by about ±0.05.