← module 10 · principal component analysis
paywalled

Analysis of a Complex of Statistical Variables into Principal Components

Harold Hotelling

Journal of Educational Psychology 24(6):417-441 and 24(7):498-520 (two parts)

about this write-up

No legal free copy was found (it may be public domain in the US if copyright was not renewed; unconfirmed). This write-up is from secondary sources and standard histories of PCA, not from a reading of the original.

the original →

the context

Psychologists had batteries of test scores (arithmetic, reading, memory and so on), and they were all related. The question was whether a few underlying quantities could explain most of the variation. The main approach, factor analysis, depended on an assumed model.

Hotelling took a more direct route: look for a few new variables, each a mix of the original ones, that capture as much of the total variation as possible.

the construction

  • The first principal component is the mix with the largest variance.
  • The second has the largest variance among mixes unrelated to the first, and so on.
  • The mixing weights are the eigenvectors of the correlation matrix, and the variances are its eigenvalues.
  • The components are unrelated to each other, and their variances add up to the total, so you can say exactly what share each one explains.

He mostly used standardised variables (each rescaled to the same spread), because test scores come in arbitrary units and the components change if you rescale a variable.

computing them by hand

Without computers, finding eigenvectors was hard work. Hotelling used a repeat-until-it- settles method: multiply a starting vector by the matrix, rescale it, and repeat until it stops changing. That gives the strongest direction. Remove it and repeat for the next. Today this is called power iteration.

how it relates to Pearson

Pearson framed it with geometry (the closest-fitting line, measuring distance across); Hotelling framed it with statistics (the largest-variance, unrelated mixes). They give the same directions. Hotelling is usually credited with the name, the statistical view and a practical way to compute it.

in this repo

  • PCA(solver="power") uses power_iteration: multiply, rescale, check, remove the found direction, repeat.
  • PCA(solver="svd") gets the same answer in a more reliable way.
  • The PCA demo shows the two agree, and why Hotelling standardised: a feature in big units takes over the first component.