module 10
Principal Component Analysis
On Lines and Planes of Closest Fit to Systems of Points in Space
Karl Pearson
Finds the straight line (or flat plane) that sits closest to a cloud of points, measuring distance straight across rather than up and down. That line is the first principal component.
Analysis of a Complex of Statistical Variables into Principal Components
Harold Hotelling
Looks for a few combinations of many measurements that capture most of the variation, each unrelated to the others. It named them principal components and gave a step-by- step way to compute them.
two questions, one answer
PCA is usually described in one sentence: "find the directions where the data varies most." The two papers behind it asked two different questions, and the surprise is that both get the same answer.
pearson: what's the closest line?
Ordinary regression of y on x measures distance straight up and down. That
treats x as exact and only y as noisy, and regressing x on y gives a different
line. Pearson pointed out that often neither variable is special. So measure distance
straight across to the line instead.
the line of closest fit
His answer: the best line goes through the middle of the cloud, along its longest direction.
hotelling: which combinations explain the most?
Hotelling had many related test scores. He looked for a few combinations of them that:
- vary as much as possible (the first component),
- then vary as much as possible while being unrelated to the first,
- and so on.
He called them principal components and gave a step-by-step way to find them.
why they agree
For any direction through the middle, each point's distance from the middle splits into two parts: the part along the line and the part across it (Pythagoras). The total is fixed. So making the "along" part as big as possible (Hotelling) is the same as making the "across" part as small as possible (Pearson).
what you get
- The components are at right angles to each other, and the new coordinates are unrelated.
- Throwing away components costs exactly the spread they held. Keep the top few and the error is the sum of what you dropped.
- Each component's share of the total spread tells you how much it matters.
how the code works
| idea | in pca.py |
|---|---|
| move the cloud to the origin | self.mean_ |
| the components, the reliable way | an SVD of the centred data |
| Hotelling's way | power_iteration: multiply and rescale until it settles, remove that direction, repeat |
| new coordinates and back again | transform, inverse_transform |
| Pearson's distance | reconstruction_error |
what the demo checks
Run uv run python examples/10_pca.py.
- Pearson's claim, by brute force. Trying 18,001 line angles finds exactly the first component's slope. Both regression lines are off to either side.
- Hotelling's step-by-step method matches the SVD to 5 × 10⁻⁷.
- Reconstruction error equals the spread that was thrown away, to four decimals.
- Images made of 3 patterns plus noise: 3 components carry almost everything, and what's left is exactly the noise.
when it goes wrong
- Unscaled features: a feature measured in grams takes over the first component.
- Curved data (a ring, a spiral): PCA can only rotate and drop directions; it can't unroll anything.
- Big spread isn't the same as important: the direction that separates two classes can have small spread and get thrown away.