Part IX — Dimensionality Reduction

Chapter 28 · Principal Component Analysis

Part IX asks the mirror-image of Part VIII's question: not more data, fewer dimensions. The Wine dataset measures each of 178 wines on 13 chemical features — yet the three grape varieties separate almost perfectly on just two axes nobody measured. Principal Component Analysis finds those axes: the directions where the data varies most. The method is built by hand on four points — covariance matrix, characteristic polynomial, eigenvectors, projections, and the exact arithmetic of what compression costs — then run for real on all 13 features, where every eigenvalue, loading, and classification score on this page comes from the actual run.

📖 Chapter 28 of 33 ⏱ ~95 min read Mostly Core Stable Volatile 📝 Quiz · 30 MCQs

🎯 Learning Objectives

State why 13 features are often 3 pieces of information wearing disguises — and what dimensionality reduction buys
Run PCA by hand on four data points: center, covariance, characteristic polynomial, eigenvectors, projections — exact arithmetic throughout
Prove the two faces of PCA: maximum projected variance and minimum reconstruction error, and that discarded eigenvalues ARE the error
Read the real Wine run: 13 genuine eigenvalues, 36.9% + 19.3% = 56.2% on two PCs, and the scree/cumulative tables
Interpret loadings — what PC1's phenolic axis and PC2's color axis mean chemically — and choose k by three criteria
Judge PCA honestly: unsupervised, variance ≠ class information (k = 2 scores 0.972, k = 3 scores 1.000 — real numbers), and the scaling trap

1. Thirteen Features, Three Wines

The running dataset: Wine.csv — 178 wines from three Italian cultivars (59 + 71 + 48), each measured on 13 chemical features: alcohol, malic acid, ash, alkalinity of ash, magnesium, total phenols, flavanoids, non-flavanoid phenols, proanthocyanins, colour intensity, hue, OD280, and proline. The task behind the task: these 13 numbers describe wines from three grapes — how much of that 13-dimensional measurement is actual information?

Chapter 12 named the disease: the curse of dimensionality — more columns mean sparser data, farther neighbours, noisier models. But wine chemistry has a deeper property, one Chapter 5 diagnosed as multicollinearity: the features correlate. Flavanoids tracks total phenols (phenols are partly flavanoids); colour intensity tracks hue; proline tracks alcohol. Thirteen columns carry far fewer than thirteen independent stories. Dimensionality reduction is the family of methods that finds the true stories and keeps only them. PCA is the founding member.

What reduction buys	Where you will see it today
Visualisation — 13-D cannot be plotted; 2-D can	Section 6: three grape varieties, visible by eye
Compression — fewer numbers to store, faster models	2 numbers per wine instead of 13
De-noising — small-variance directions are often noise	Section 6: last 6 PCs carry under 9% of variance
Decorrelation — PCs are uncorrelated by construction	Section 3: the eigenvectors are orthogonal
Collinearity cure — regression on PCs has no VIF problem	Chapter 5's 50_Startups diagnosis, resolved in principle

📝

Note: throughout Part IX the pipeline splits the data first (142 train / 36 test here, seed 0) and standardises on the training set only — Chapter 3's leakage rule. Why scaling is not optional here becomes the chapter's loudest warning.

2. The Idea: Variance is Information

Picture the wines as a cloud of 178 points in 13-dimensional space. Now photograph that cloud from the one direction that shows it most spread out — that photograph is the first principal component. Photograph it again from the perpendicular direction with the next-most spread: the second component. Repeat, and you have a new coordinate system, ordered by how much of the data's scatter each axis captures.

The working hypothesis is audacious and usually right: the directions along which data varies most are where the information lives. Directions where every wine measures nearly the same tell you nothing that distinguishes them. PCA formalises "spread out" as variance, and hunts for the axes of maximal variance.

Why not simply delete the least-useful column? Because that throws information away blindly. A column can be individually weak yet decisively useful in combination — Chapter 11's SVM and Chapter 14's kernels existed precisely because combinations beat individual features. PCA instead builds new axes as weighted combinations of all features — every PC is a recipe using all 13 ingredients in optimized proportions — and keeps only the recipes that matter.

💡

Pro Tip — two equivalent definitions: the first PC is simultaneously (a) the projection direction with maximum variance of the projected data, and (b) the line with minimum squared reconstruction error when the data is compressed onto it. Section 3 verifies both numerically — they are the same theorem seen from two sides.

3. PCA by Hand — Four Points, Two Axes

Four wines, two features (say, acidity and sweetness — units don't matter to the arithmetic): (3, 3), (7, 7), (6, 4), (4, 6). Every step below is exact, and verified against the library at the end.

3.1 Center

PCA measures deviations, so first subtract the mean (5, 5). The centered data: (−2, −2), (2, 2), (1, −1), (−1, 1). The cloud now tilts along the diagonal — the eye already sees the answer; PCA will compute it.

3.2 The covariance matrix

C = (1/n) · ZᵗZ =

2.5	1.5
1.5	2.5

var(x) = (4+4+1+1)/4 = 2.5 · var(y) = 2.5 the same way · cov = (−2·−2 + 2·2 + 1·−1 + −1·1)/4 = (4+4−1−1)/4 = 1.5. The off-diagonal 1.5 > 0 is the tilt: x and y move together.

3.3 Eigendecomposition — the axes fall out

Every symmetric matrix has perpendicular eigenvectors, and PCA's whole claim is that the eigenvectors of the covariance matrix are the principal axes. Solve by hand with the characteristic polynomial:

det(C − λI) = (2.5 − λ)² − 1.5² = λ² − 5λ + 4 = (λ − 4)(λ − 1) = 0

λ₁ = 4 and λ₂ = 1. Eigenvectors: for λ₁ = 4, (C − 4I)v = 0 gives −1.5v₁ + 1.5v₂ = 0 → v₁ = v₂ → v₁ = (1, 1)/√2 ≈ (0.707, 0.707) — the 45° diagonal. For λ₂ = 1: 1.5v₁ + 1.5v₂ = 0 → v₂ = (1, −1)/√2 — perpendicular, as promised.

3.4 Project: the scores

Each point's new coordinates are dot products with the eigenvectors. The score on PC1 = (x + y)/√2 after centering:

Point (centered)	PC1 score = (x+y)/√2	PC2 score = (x−y)/√2
(−2, −2)	−4/√2 = −2.83	0
(2, 2)	+4/√2 = +2.83	0
(1, −1)	0	+2/√2 = +1.41
(−1, 1)	0	−2/√2 = −1.41

Figure 28.1 — The hand example. PC1 (solid) runs along the diagonal — maximum spread; PC2 (dashed) is perpendicular. Point (6, 4) has PC1 score 0: it sits exactly at the mean along the main axis.

3.5 The variance ledger — and what compression costs

The eigenvalues are the variances of the scores. Check: mean of PC1² = (8 + 8 + 0 + 0)/4 = 4 = λ₁; mean of PC2² = (0 + 0 + 2 + 2)/4 = 1 = λ₂. Two consequences, both exact:

explained variance PC1 = λ₁ / (λ₁ + λ₂) = 4/5 = 80% · PC2 = 20%

compression to PC1: average squared reconstruction error = λ₂ = 1

Verify the error claim on (6, 4): its PC1 score is 0, so reconstruction = mean + 0 = (5, 5); the error vector is (1, −1), squared length 2. Across all four points the squared errors are 0, 0, 2, 2 — average 1, exactly the discarded eigenvalue. The variance you throw away is the error you eat, coin for coin. (Library check: sklearn on the same four points returns EVR 80/20 and identical reconstructions.)

⚠

Warning — sign is a costume: an eigenvector and its negative are the same axis; sklearn may return (−0.707, −0.707) where the hand computation gives (0.707, 0.707), flipping score signs. Nothing has changed — the axis, variance, and reconstructions are identical. Sign ambiguity is a classic PCA gotcha when comparing implementations.

4. The Algorithm, and What the Eigenvalues Mean

The complete recipe, exactly as executed by hand above:

Center each feature (subtract its mean) — PCA is about deviations.
Scale each feature to unit variance — otherwise big-unit features hijack the axes (Section 6 shows how badly).
Covariance matrix C — p × p for p features; captures every feature's spread and every pair's co-movement.
Eigendecompose C = VΛVᵗ: eigenvalues λᵢ (variances along each axis), eigenvectors (the axes, mutually perpendicular).
Sort by λ descending — PC1, PC2, …
Project: scores T = ZV; keep the first k columns to compress.

Three structural facts make the method feel engineered rather than lucky. Conservation: the eigenvalues sum to the trace of C — total variance is preserved, merely repackaged by axis. On the scaled Wine data, 13 features each carry variance 1, total 13; the real eigenvalues of Section 6 sum to exactly that. Orthogonality: the PCs are uncorrelated by construction — the collinearity disease of Section 1 is cured outright, each PC a genuinely independent story. Best rank-k: keeping the top k axes is provably the rank-k compression with minimum squared error — not a heuristic.

📝

Note — SVD in practice: software does not form the 13 × 13 covariance matrix and factorise it; it applies the singular value decomposition directly to the centered data matrix — numerically stabler, same result (singular values are the square roots of the eigenvalues, up to the 1/n scaling). The hand route and the library route meet at the same axes.

📋 Stable content — Reviewed: October 2026

5. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Wine.csv')
X = dataset.iloc[:, :-1].values      # 13 chemical features
y = dataset.iloc[:, -1].values       # Customer_Segment: 1, 2, 3

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)  # scale on train ONLY
X_test = sc.transform(X_test)

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_train = pca.fit_transform(X_train) # fit PCA on train ONLY
X_test = pca.transform(X_test)
print(pca.explained_variance_ratio_)  # [0.369, 0.193]

from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state=0)
classifier.fit(X_train, y_train)

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
print(confusion_matrix(y_test, y_pred))   # [[14  0  0]
                                          #  [ 1 15  0]
                                          #  [ 0  0  6]]
print(accuracy_score(y_test, y_pred))     # 0.972

Listing 28.1 — The complete pipeline (source: principal_component_analysis.py). Scaling and PCA both fit on the training set only — the test set is transformed with frozen parameters.

# The idioms worth knowing

PCA(n_components=0.95)   # keep 95% of variance — k chosen automatically
PCA().fit(X)             # full decomposition: explained_variance_, components_

# leakage-proof one-liner with a Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
pipe = Pipeline([('scale', StandardScaler()),
                 ('pca', PCA(n_components=2)),
                 ('clf', LogisticRegression(random_state=0))])
pipe.fit(X_train_raw, y_train)          # correct order, guaranteed

Listing 28.2 — Variance-targeted component choice and the Pipeline idiom.

💡

Pro Tip: PCA(n_components=0.95) is the professional default when you don't want to argue about k: hand it a variance target instead of a count. On this data it keeps 10 components (Section 7's table shows why).

6. The Real Run — Wine in Two Dimensions

The real decomposition of the 13 scaled training features — every eigenvalue below from the actual fit:

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10	PC11	PC12	PC13
λ (eigenvalue)	4.83	2.53	1.41	0.97	0.82	0.64	0.54	0.33	0.30	0.24	0.23	0.16	0.09
Variance %	36.9	19.3	10.8	7.4	6.3	4.9	4.1	2.5	2.3	1.9	1.7	1.3	0.7
Cumulative %	36.9	56.2	67.0	74.4	80.6	85.5	89.7	92.2	94.5	96.3	98.1	99.3	100

The ledger balances perfectly: the eigenvalues sum to 13.00 — total variance conserved, exactly as Section 4 promised. And the shape of the table is the classic PCA signature: two strong components (36.9% + 19.3% = 56.2%), a shoulder at PC3, then a long shallow tail — the last six axes together carry under 9% of the variance, mostly noise and rounding. Thirteen measurements, but roughly three dimensions of real chemistry — the redundancy Section 1 suspected, now quantified.

6.1 The two-dimensional photograph

Project all 142 training wines onto the first two PCs — the scatter the source's contour plots are famous for, drawn here from the real coordinates:

Figure 28.2 — All 142 training wines, real PC coordinates. Cultivar 1 (red) sits high-right, cultivar 2 (green) low-centre, cultivar 3 (blue) left. × marks each class mean: (2.39, 1.00), (0.09, −1.68), (−2.67, 1.13). Three clusters — and PCA never looked at the labels.

The punchline of the whole part: PCA is unsupervised — the class labels were never shown to it — yet the three cultivars separate into three visible islands. The grape's identity was hiding inside the chemistry's variance structure all along.

6.2 Classify on two numbers

The source completes the loop with logistic regression (Chapter 11, multiclass) on the two PC scores — train, then the 36-wine test set:

Model (same split, same scaling)	Confusion matrix	Accuracy	Features used
Logistic on PC1 + PC2	[[14, 0, 0], [1, 15, 0], [0, 0, 6]]	0.972	2 (56.2% of variance)
Logistic on all 13 scaled features	[[14, 0, 0], [0, 16, 0], [0, 0, 6]]	1.000	13 (100%)
Logistic on PC1 only	—	0.806	1 (36.9%)
Logistic on PCs 1–3	—	1.000	3 (67.0%)
Logistic on PCs 1–5	—	0.972	5 (80.6%)
Logistic on PCs 1–7	—	1.000	7 (89.7%)

Read this table twice, because it teaches two opposite lessons. First: two numbers per wine — 15% of the columns, 56% of the variance — classify 35 of 36 test wines correctly. The compression is real. Second — and subtler: accuracy does not track variance. Three PCs (67%) already reach the perfect 13-feature score; five PCs (80.6%) score lower than two. Maximum-variance directions are usually where information lives, but a low-variance direction can still carry class signal, and extra PCs import noise. Variance is a heuristic for information, not a proof. (And with only 36 test wines, one wine = 2.8 percentage points — Chapter 18's small-sample caution applies.)

⚠

Warning — the scaling trap: skip the StandardScaler and PCA on raw Wine data would be proline-analysis: proline's variance (std 314.9) is over a million times hue's (std 0.23), so PC1 would point almost entirely along the proline axis — not because proline is informative, but because it is measured in big numbers. Chapter 12's KNN lesson, now with eigenvectors. Standardise first, always.

7. Choosing k & Reading Loadings

7.1 Three ways to pick the number of components

Criterion	Rule	On this data
Scree elbow	plot λ vs PC; keep components before the curve flattens	elbow at PC2–PC3 → keep 2–3
Cumulative threshold	smallest k reaching a target variance	80% → 5 · 90% → 8 · 95% → 10 · 99% → 12
Kaiser criterion	keep PCs with λ > 1 (more variance than one original feature)	λ = 4.83, 2.53, 1.41 → 3 components

All three point the same way here: two or three components. Notice the satisfying echo: Kaiser's λ > 1 rule selects exactly the three components that delivered the perfect 1.000 test accuracy — and the scaled data's "one original feature" benchmark of variance 1 is what makes the rule's logic visible (a PC must out-earn a single feature to be worth keeping).

7.2 Loadings — what the axes mean

Each PC is a weighted recipe over the original features; the weights are the loadings. The real values, strongest six per component:

PC1 loadings	value	PC2 loadings	value
Flavanoids	+0.428	Colour intensity	+0.531
Total phenols	+0.389	Alcohol	+0.498
OD280	+0.376	Proline	+0.365
Proanthocyanins	+0.308	Ash	+0.315
Hue	+0.307	Hue	−0.272
Non-flavanoid phenols	−0.305	Magnesium	+0.258

The chemistry reads off the page. PC1 is a phenolic-quality axis: heavy positive weights on flavanoids, total phenols, OD280 (a phenolic-dilution measure) and proanthocyanins, negative on non-flavanoid phenols — wines rich in "good" phenolics score high, wines with the "lesser" phenolics score low. PC2 is a body/colour axis: colour intensity, alcohol and proline up — big, strong, dark wines high. A wine's two PC scores are, roughly, "phenolic character" and "body" — two summary measurements a wine chemist would endorse, discovered from correlation structure alone.

💡

Pro Tip: loadings are also the debugging tool when PCA misbehaves: if one feature dominates a loading vector almost alone, you forgot to scale (or that feature genuinely is the story — the loadings cannot tell you which; domain knowledge must).

⚠ Volatile content — Reviewed: October 2026 · Next review: January 2027

8. Limitations & the Road Ahead

The honest ledger — assumptions and failure modes first:

Linearity: PCA finds straight axes through the cloud. If the data lies on a curve (Chapter 14's moon shapes), the leading PCs cut across it. Kernel PCA — Chapter 30 — lifts PCA into feature space exactly as the kernel trick lifted SVM.
Variance ≠ class information: PCA never sees y. This chapter's own table is the exhibit: 5 PCs score 0.972 while 3 PCs score 1.000. When labels exist, the supervised reducer — LDA, Chapter 29 — aims axes directly at class separation, and typically needs fewer components still (LDA caps at classes − 1 = 2 here).
Interpretability tax: PC1 is 13 numbers in a trench coat. Loadings recover meaning (Section 7.2), but a model built on scores explains itself less readily than one built on milligrams — the regulator's question from Chapter 26, reprised.
Outlier sensitivity: variance is a squared quantity; one wild wine can rotate the leading axis toward itself. Inspect the PC-score extremes before trusting a decomposition.
The modern neighbourhood (the volatile part): t-SNE and UMAP dominate visual exploration of high-dimensional data — better cluster pictures, but non-linear, parameter-heavy, and unsuitable for the "then classify on it" pipeline PCA serves so well; autoencoders (Chapter 26's machinery) learn non-linear compressions at the cost of much more data and compute. For a first, honest, invertible compression of correlated features, PCA remains the professional default.

Next chapter: Linear Discriminant Analysis — the same eigen-machinery, aimed by the labels. Where PCA asks "along which direction do wines differ most from each other?", LDA asks "along which direction do the three cultivars differ most from each other?" One word of difference in the question; a different matrix in the computation; and, usually, a sharper separator.

🛠 Hands-On: The Sommelier's Two NumbersIntermediate

A wine retailer wants each wine catalogued by two summary scores instead of 13 lab measurements. Build the transformation, verify every hand claim, then advise: are two numbers enough, and what do they mean?

Steps

Hand-verify Section 3 completely: covariance entries, characteristic polynomial (λ−4)(λ−1), eigenvectors, all eight scores, the 80/20 split, and the reconstruction of (6, 4) → (5, 5) with squared error 2.
Confirm the hand result against sklearn on the same four points: PCA(2) EVR must be [0.8, 0.2] and the reconstruction from PC1-only must be [(3,3), (7,7), (5,5), (5,5)]. Explain any sign differences.
Run the Wine pipeline (Listing 28.1); verify the eigenvalue table's first three entries (4.83, 2.53, 1.41) and their sum-to-13 conservation.
Reproduce the k-sweep (k = 1, 2, 3, 5, 7, 10, 13) and the two headline accuracies: 0.972 on two PCs, 1.000 on thirteen.
Choose k by all three criteria of Section 7.1 and defend a recommendation to the retailer in three sentences.
Bonus: rerun PCA WITHOUT StandardScaler; report PC1's explained variance and top loading, and explain what the axis now measures.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# ---- steps 1-2: hand example vs library
pts = np.array([[3,3],[7,7],[6,4],[4,6]], float)
p = PCA(n_components=2).fit(pts)
print(p.explained_variance_ratio_)        # [0.8 0.2]
T = p.transform(pts)
T1 = T.copy(); T1[:,1] = 0                # keep PC1 only
print(p.inverse_transform(T1))            # [[3,3],[7,7],[5,5],[5,5]]

# ---- steps 3-5: the Wine pipeline
ds = pd.read_csv('Wine.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.2, random_state=0)
sc = StandardScaler(); Xtr_s = sc.fit_transform(Xtr); Xte_s = sc.transform(Xte)
full = PCA().fit(Xtr_s)
print(full.explained_variance_[:3].round(3))   # [4.829 2.529 1.408]
print(full.explained_variance_.sum().round(3)) # 13.0 — conservation
for k in [1,2,3,5,7,10,13]:
    pk = PCA(n_components=k).fit(Xtr_s)
    acc = LogisticRegression(random_state=0).fit(
        pk.transform(Xtr_s), ytr).score(pk.transform(Xte_s), yte)
    print('k=%2d  cum var %5.1f%%  acc %.3f'
          % (k, full.explained_variance_ratio_[:k].sum()*100, acc))

# ---- step 6 (bonus): the scaling trap, witnessed
raw = PCA(2).fit(Xtr)                     # NO scaler
print(raw.explained_variance_ratio_)      # PC1 ≈ 0.998
top = np.abs(raw.components_[0]).argmax()
print('PC1 is dominated by:', ds.columns[top])   # Proline

Expected results (real): hand/library agree exactly (EVR 80/20; reconstruction to (5,5) twice; mean squared error 1 = λ₂). Wine: eigenvalues 4.829, 2.529, 1.408, … summing to 13.000; k-sweep acc 0.806 (k=1), 0.972 (k=2), 1.000 (k=3), 0.972 (k=5), 1.000 (k=7), 1.000 (k=10), 1.000 (k=13). Without scaling, PC1 captures ≈99.8% of variance and points almost entirely along Proline — the milligram scale, not the chemistry. Recommendation sketch: catalogue by two scores (phenolic character, body — 56.2% of variance, 0.972 downstream accuracy, human-readable loadings), keep the third component in reserve since Kaiser's rule and the k=3 accuracy both endorse it.

🎯 Key Takeaways

Correlated features overcount the same information: 13 wine measurements carry ~3 real dimensions. PCA repackages variance into perpendicular axes ranked by importance.

The eigenvectors of the covariance matrix ARE the principal axes — hand-verified: λ² − 5λ + 4 = 0 → λ = 4, 1 on the 45° diagonal, 80%/20% split.

Discarded eigenvalues are the compression bill: keeping PC1 only, mean squared error = λ₂ = 1, exactly. Max variance and min error are one theorem.

Real Wine run: PC1 36.9% + PC2 19.3% = 56.2%, eigenvalues sum to 13.00; three cultivars separate visibly — and PCA never saw the labels.

Two numbers classify 35/36 test wines (0.972); three reach 1.000 — but five drop back to 0.972. Variance is a heuristic for information, not a guarantee.

Scale first — unscaled Wine makes PCA "proline-analysis" (PC1 ≈ 99.8% one feature). Loadings make axes readable: PC1 = phenolic quality, PC2 = body/colour.