module 10

Principal Component Analysis

free pdf

On Lines and Planes of Closest Fit to Systems of Points in Space

Karl Pearson

Finds the straight line (or flat plane) that sits closest to a cloud of points, measuring distance straight across rather than up and down. That line is the first principal component.

paywalled

Analysis of a Complex of Statistical Variables into Principal Components

Harold Hotelling

Looks for a few combinations of many measurements that capture most of the variation, each unrelated to the others. It named them principal components and gave a step-by- step way to compute them.

two questions, one answer

PCA is usually described in one sentence: "find the directions where the data varies most." The two papers behind it asked two different questions, and the surprise is that both get the same answer.

pearson: what's the closest line?

Ordinary regression of y on x measures distance straight up and down. That treats x as exact and only y as noisy, and regressing x on y gives a different line. Pearson pointed out that often neither variable is special. So measure distance straight across to the line instead.

the line of closest fit

-2024-3-2-10123
the first principal component: the line through the middle that keeps the perpendicular distances (thin lines) as short as possible. it follows the longest direction of the cloud and holds 86% of the spread.

His answer: the best line goes through the middle of the cloud, along its longest direction.

hotelling: which combinations explain the most?

Hotelling had many related test scores. He looked for a few combinations of them that:

  • vary as much as possible (the first component),
  • then vary as much as possible while being unrelated to the first,
  • and so on.

He called them principal components and gave a step-by-step way to find them.

why they agree

For any direction through the middle, each point's distance from the middle splits into two parts: the part along the line and the part across it (Pythagoras). The total is fixed. So making the "along" part as big as possible (Hotelling) is the same as making the "across" part as small as possible (Pearson).

what you get

  • The components are at right angles to each other, and the new coordinates are unrelated.
  • Throwing away components costs exactly the spread they held. Keep the top few and the error is the sum of what you dropped.
  • Each component's share of the total spread tells you how much it matters.

how the code works

idea in pca.py
move the cloud to the origin self.mean_
the components, the reliable way an SVD of the centred data
Hotelling's way power_iteration: multiply and rescale until it settles, remove that direction, repeat
new coordinates and back again transform, inverse_transform
Pearson's distance reconstruction_error

what the demo checks

Run uv run python examples/10_pca.py.

  • Pearson's claim, by brute force. Trying 18,001 line angles finds exactly the first component's slope. Both regression lines are off to either side.
  • Hotelling's step-by-step method matches the SVD to 5 × 10⁻⁷.
  • Reconstruction error equals the spread that was thrown away, to four decimals.
  • Images made of 3 patterns plus noise: 3 components carry almost everything, and what's left is exactly the noise.

when it goes wrong

  • Unscaled features: a feature measured in grams takes over the first component.
  • Curved data (a ring, a spiral): PCA can only rotate and drop directions; it can't unroll anything.
  • Big spread isn't the same as important: the direction that separates two classes can have small spread and get thrown away.