The oldest ideas in dimensionality reduction are still, often, the right ones. PCA, ICA, NMF, and kernel PCA are all linear projections, differing only in what they consider a good projection to be.
Linear dimensionality reduction is a matrix multiplication. Given data X with n rows and d columns, pick a skinnier matrix W with d rows and k columns, and the embedding is XW. That is it. The whole family of methods we cover here differs only in how they pick W.
That sounds like a narrow design space, and it is, but the narrowness is exactly why linear methods are still useful. They are fast, deterministic, composable, and interpretable. When a nonlinear method distorts your data in some hard-to-describe way, it is often linear methods that serve as the baseline you compare against. And when a nonlinear method needs a sensible starting configuration, it is often PCA that provides it.
Principal Component Analysis, introduced by Pearson in 1901 and independently by Hotelling in 1933, asks a simple question: among all possible directions through the data, which one has the most spread? That direction becomes the first principal component. The second is the direction of most spread orthogonal to the first. The third is the direction of most spread orthogonal to both. Keep going, order by spread, keep the top few.
Geometrically, the first few components define a subspace that lies as close as possible to the data in a least-squares sense. If you were asked to summarize the cloud with a line, you would pick the first component. With a plane, the first two.
As the cloud grows more anisotropic, the first component captures a larger share of the variance. In the extreme case where the cloud is a perfect line, PC1 captures 100% and a one-dimensional embedding loses nothing. Most real data is somewhere in between: the first few components capture most of the variance, and the rest is treated as noise.
PCA has a clean mathematical life. The components are eigenvectors of the data's covariance matrix; equivalently, the right singular vectors of the centered data. Modern implementations use SVD because it is numerically more stable. Explained variance is literally the eigenvalue, so picking a number of components reduces to choosing a cutoff on a sorted list.
What PCA cannot do is bend. If the meaningful structure of the data lies on a curved manifold, no linear subspace is going to be a good fit. This is the observation that motivates almost every nonlinear method in this series. PCA sees a swiss roll as an elongated blob; it has no way to unroll it. But PCA is often the right baseline, often the right initialization, and, for data that genuinely has linear structure (and a lot of data does, or close to it) often the right answer.
Imagine three microphones in a room with three people talking. Each microphone picks up a mixture of all three voices, with the mixing weights depending on distances and geometry. Can we recover the individual voices from just the mixtures?
PCA cannot do this. The principal components of the mixtures are the directions of highest variance, which have no reason to align with the individual speakers. PCA treats correlation, not the identity of the underlying sources.
Independent Component Analysis, developed through the 1990s and codified by Hyvärinen and Oja's FastICA in 2000, attacks the same problem from a different angle. Its core assumption: the original sources are statistically independent, and most interesting real-world signals are non-Gaussian. A mixture of independent non-Gaussian signals is closer to Gaussian than any of the sources on their own (this is a consequence of the central limit theorem). So if we search among the directions in the data that are the least Gaussian, we find the directions of the original sources.
The key intuition is that the Gaussian distribution is the enemy. If all your sources were Gaussian, you could not recover them: a rotation of Gaussian noise is still Gaussian noise, so there is no way to pick out the original orientation. Non-Gaussianity is what makes the problem solvable.
ICA is a workhorse in neuroimaging: EEG and fMRI produce signals that are linear mixtures of underlying brain sources plus artifacts (eye blinks, heartbeat, muscle activity). Running ICA and manually labeling components is a standard preprocessing step. In machine learning, ICA is less commonly used as a front-line DR method, but it shows up in feature engineering, denoising, and as a tool for analyzing learned representations.
Suppose your data are pixel intensities, word counts, gene expression levels, or anything else that cannot physically be negative. PCA is happy to write your face as "average face + 0.7 times (component 1, which has negative regions) minus 0.3 times (component 2)." The math is fine. The interpretation is weird: what does it mean for a face to contain a negative eye?
Non-negative Matrix Factorization, formalized by Lee and Seung in 1999, adds a simple constraint: both the basis patterns and the coefficients must be non-negative. Each sample is a sum (never a difference) of non-negative parts. In Lee and Seung's original face paper, the discovered basis looks like localized face features: noses, eyebrows, mouths. Every face is a positive combination of them. The constraint buys interpretability.
NMF has become essential in single-cell genomics, where cells are linear combinations of gene expression programs; in topic modeling, where documents are combinations of word-distribution topics; and in audio source separation, where spectrograms are sums of non-negative frequency patterns. Wherever data has a natural "parts" reading, NMF tends to find the parts.
The tradeoffs: the factorization is not unique (many valid decompositions of the same data exist), the optimization is non-convex, and picking the rank is an art. But when the data is genuinely non-negative and the parts interpretation is meaningful, NMF often produces a decomposition that downstream analysts can actually reason about, which is rare in DR.
PCA assumes the structure in the data is linear. What if it is not, but could be made linear by a clever change of coordinates? This is the idea that Schölkopf, Smola, and Müller published in 1998 under the name kernel PCA.
Imagine a dataset shaped like two concentric circles. No linear projection separates them, because their separation is not along any direction; it is along the radius. But if you map each point (x, y) to (x, y, x² + y²), the two circles now lie at different heights, and they are linearly separable in the new coordinate system. Linear methods in the transformed space correspond to nonlinear methods in the original space.
The catch is that writing down the transformation explicitly becomes expensive in high dimensions (and, for some useful transformations, infinite). The kernel trick sidesteps this: PCA only needs inner products between points, and an inner product in the transformed space is computable directly from a kernel function k(x, y) applied to points in the original space. So we never build the transformed space; we just compute a Gram matrix of kernel values and do PCA on that.
Common kernels are the polynomial kernel (captures pairwise interactions) and the RBF or Gaussian kernel (captures local similarity, softly). The choice of kernel and its hyperparameters are the knobs the user turns. Kernel PCA does recover nonlinear structure in many classical synthetic examples, and it turns out to be a special case of a much broader family: Laplacian eigenmaps, Isomap, diffusion maps, and spectral clustering can all be viewed as kernel methods with particular kernels. We will see them in the next article.
What holds kernel PCA back in practice is scale: the Gram matrix is n-by-n, and eigendecomposing it is cubic in n. Beyond a few thousand points, it becomes unworkable without approximation. For large data, the neighbor-based methods we cover in article 4 have won out.
The last member of the linear family is the most surprising. Instead of carefully choosing W to preserve some property, just draw W at random. Gaussian entries, or even signed sparse entries work fine. Multiply. You are done.
This sounds like it should not work, and yet it does. The Johnson-Lindenstrauss lemma proves that for any desired precision, a random projection into O(log(n) / ε²) dimensions approximately preserves all pairwise distances in the dataset, regardless of the original dimension. Not because the random projection is smart, but because high-dimensional geometry itself is concentrated enough that a random direction is almost always a reasonable summary.
Random projection is the go-to when you want to crush dimension for speed, not for visualization. As preprocessing before clustering, before nearest-neighbor search, or before handing data to a more expensive method, a random projection from 10,000 to 100 dimensions is cheap to compute, provably loses little, and has no hyperparameters to tune. The methods in the rest of this series often use it as preprocessing without advertising the fact.
Across this family, the organizing principle is clear: a good projection is one where a linear combination of coordinates captures the property we care about, whether that is variance (PCA), independence (ICA), additive parts (NMF), kernelized similarity (kernel PCA), or just distance (random projection). The design space is a matrix multiplication, and the methods differ in their objectives and constraints.
Linear methods struggle when the structure is fundamentally curved. A swiss roll, two intertwined crescents, a branching developmental trajectory: these are all cases where the local geometry does not match the global geometry, and no single linear projection can reconcile them. The next article picks up there, with methods that replace the Euclidean distance between points with something curvier: distances along the manifold, distances through a graph, distances from a random walk.