Part IX — Dimensionality Reduction

Chapter 29 · Linear Discriminant Analysis

Chapter 28 asked the data "where do you vary most?" and the answer — two principal components — classified 35 of 36 wines. This chapter changes one word in the question and asks the labels too: "where do the three cultivars vary most — beyond their own noise?" Linear Discriminant Analysis is PCA's supervised twin: the same eigendecomposition machinery, aimed at class separation by Fisher's criterion. Fisher's ratio is computed by hand on an eight-point dataset — between-class gap over within-class spread, every number exact — and then the real Wine run reduces 13 features to 2 discriminants on which the classifier scores a perfect 36 of 36.

📖 Chapter 29 of 33 ⏱ ~90 min read Mostly Core Stable Volatile 📝 Quiz · 30 MCQs

🎯 Learning Objectives

State the one-word difference from PCA — supervised reduction — and what it buys (and costs)
Derive Fisher's criterion J(w) = between-class separation / within-class spread, and explain why noisy directions get discounted
Run LDA by hand on eight points: class means, within-class scatter SḦ, the optimal direction w ∝ SḦ⁻¹(m₂−m₁), and the J-values of four candidate directions
Explain the classes − 1 ceiling on components, and LDA's second life as a classifier (Gaussian classes, shared covariance)
Read the real Wine run: LD1's 73.9% of between-class variance, the class ranges, and the perfect 1.000 test score versus PCA's 0.972
Judge PCA versus LDA honestly — when the labels help, when they hurt, and what breaks each

1. One Word Different

Same dataset as Chapter 28 — 178 wines, 13 chemical features, three cultivars (59 + 71 + 48), split 142 train / 36 test with seed 0, standardised on the training set. Same pipeline shape: reduce, then classify with logistic regression. One change, visible in the single line that differs between the two chapters' source code:

X_train = pca.fit_transform(X_train)          # Chapter 28 — unsupervised
X_train = lda.fit_transform(X_train, y_train) # Chapter 29 — supervised

Listing 29.1 — The entire difference between PCA and LDA, in one argument.

PCA hunted variance blind to the labels; LDA is handed them. The consequence runs both ways. Up side: the new axes are built to separate the classes, so fewer components do more work — Chapter 28 needed 56.2% of the variance to score 0.972; this chapter's two discriminants score 1.000. Down side: supervision is a loan, repaid in assumptions — LDA believes the classes are (roughly) Gaussian blobs of equal spread, it cannot exceed classes − 1 components, and directions that separate the training labels too well may be memorising noise. Section 7 prices both sides of the trade.

📝

Note: "Linear Discriminant Analysis" names two things at once — a dimensionality reducer (what this chapter does with it, PCA-style) and a classifier in its own right (Section 4). Both are the same mathematics wearing different hats; the chapter runs both, for real.

2. Fisher's Criterion

Picture two classes as clouds. A good separation direction is one where the clouds' centres land far apart after projection — and the clouds themselves stay tight. Far apart alone is not enough: two centres separated along a direction in which both clouds are smeared will still overlap after projection. Ronald Fisher's 1936 insight (the same year he introduced the iris data) was to put both desires in one fraction and maximise it:

J(w) = (projected separation)² / projected within-class spread

Numerator — between-class: how far the projected class means land from each other, weighted by class size. Denominator — within-class: how much each class is smeared along w, summed over classes. Maximising J pushes classes apart relative to their own noise — a direction where centres separate by 4 units of noise is worth more than one where they separate by 5 units of noise.

The two ingredients have names that will follow the rest of the chapter. The between-class scatter Sఖ measures class-mean dispersion around the grand mean. The within-class scatter SḦ is the sum, over classes, of each class's own scatter around its own mean — noise, aggregated. Fisher's problem, maximise J(w), has a famous closed-form answer:

w∗ ∝ SḦ⁻¹ (m₂ − m₁)

mᵢ = class means. Read it as a sentence: take the mean-gap, then discount it by the noise in each direction — the inverse scatter matrix SḦ⁻¹ shrinks directions in which the classes themselves are spread out. For many classes, the same idea generalises to an eigenproblem SḦ⁻¹Sఖ w = λw — Chapter 28's machinery, one matrix swapped.

💡

Pro Tip — why "Linear" is in the name: the separator is a projection (a linear operation), and with equal class covariances the resulting decision boundaries are straight lines, planes, hyperplanes. Chapter 15's Naive Bayes produced a quadratic boundary precisely because its per-class variances differed; LDA's equal-spread assumption is what buys the linearity.

3. LDA by Hand — Eight Points, One Direction

Two classes, two features, four points each — every number below exact and verified against the library at the end:

Class A (×4)	Class B (×4)
(0, 2) · (4, 2) · (2, 1) · (2, 3)	(4, 4) · (8, 4) · (6, 3) · (6, 5)
mean m₁ = (2, 2)	mean m₂ = (6, 4)

Both clouds are elongated in the x-direction and tight in y — a deliberate design. The mean gap is (4, 2): centres sit 4 apart in x, only 2 apart in y. The naive guess is "separate along x". Fisher will disagree, and the reason is the noise.

3.1 The within-class scatter

Center each class around its own mean, sum up the squared deviations (the class scatter matrices), and add. Class A centered: (−2, 0), (2, 0), (0, −1), (0, 1) — and class B, having the same shape, centers to the same four vectors. Each class contributes Σx² = 4+4+0+0 = 8, Σy² = 0+0+1+1 = 2, Σxy = 0:

SḦ = S₁ + S₂ =

16	0
0	4

The classes are 4× noisier in x (16) than in y (4) — the setup's whole point. A gap measured in x-units is worth less than the same gap measured in y-units, because x-gaps must shout over x-noise.

3.2 The optimal direction

w∗ ∝ SḦ⁻¹ (m₂ − m₁) = (4/16, 2/4) = (0.25, 0.5) ∝ (1, 2)

The x-gap of 4 is discounted by 16-fold noise to 0.25; the y-gap of 2 is discounted by only 4-fold noise to 0.5. Despite the raw gap favouring x two-to-one, the noise reverses the verdict two-to-one: the discriminant axis climbs at slope 2. (Library check: sklearn's fitted direction has y/x ratio exactly 2.0.)

3.3 Judge four candidate directions

Fisher's criterion makes the competition quantitative — J(w) = (w·gap)² / (wᵗ SḦ w), scale-invariant, so any length of a direction scores the same:

Direction w	Numerator (w·gap)²	Denominator wᵗSḦw	J(w)	Verdict
(1, 0) — "the x-gap"	4² = 16	16	1.00	gap eaten by noise
(0, 1) — "the y-gap"	2² = 4	4	1.00	same ratio, less of both
(1, 1) — naive compromise	6² = 36	16 + 4 = 20	1.80	better…
(1, 2) — Fisher's	8² = 64	16 + 4·4 = 32	2.00	maximum

Now project the eight points onto the unit axis (1, 2)/√5 — the score of point (x, y) is (x + 2y)/√5:

	Projected scores				Band
Class A	1.79	3.58	1.79	3.58	1.8 – 3.6
Class B	5.37	7.16	5.37	7.16	5.4 – 7.2

One number per point, and the classes land in disjoint bands (A tops out at 3.58, B bottoms at 5.37 — a gap of 1.8). Along the raw x-axis, by contrast, A spans 0–4 and B spans 4–8: touching at 4. The noise-aware direction separates what the naive one merely orders. That is the entire value proposition of LDA, in eight points.

💡

Pro Tip — the Chapter 13 echo: "discount the gap by the noise" is exactly the SVM's margin logic stated in statistics: separation quality is a ratio, not a difference. Fisher (1936) and maximum-margin (1990s) arrive at the same principle from opposite ends.

4. The Algorithm & LDA's Second Life as a Classifier

The reduction algorithm, multi-class

Standardise (Chapter 28's scaling trap applies verbatim).
Class means mᵢ and the grand mean m; class counts nᵢ.
Within-class scatter SḦ — the sum over classes of each class's scatter around its own mean: the noise.
Between-class scatter Sఖ = Σᵢ nᵢ (mᵢ − m)(mᵢ − m)ᵗ — the signal.
Eigendecompose SḦ⁻¹Sఖ: eigenvalues are separation strengths, eigenvectors the discriminant axes LD1, LD2, …
Project the data onto the top axes. At most C − 1 axes carry signal — with C = 3 cultivars, two discriminants and no more.

Why the C − 1 ceiling? Between-class scatter is built from C class means, and C points in any space carry only C − 1 independent directions of deviation from their centre. Once LD1 and LD2 span everything three means can express, further axes see zeros. (Two classes → one discriminant; the boundary normal itself.)

The classifier hiding inside

LDA's second identity: assume each class is a Gaussian cloud and all clouds share one covariance matrix. Bayes' rule (Chapter 15's machinery, all features correlated this time) then says: classify to the class whose mean is closest in noise-discounted distance — Mahalanobis distance, SḦ⁻¹ again — plus a prior correction. Equal covariances collapse the quadratic terms, and the decision boundaries come out straight. This is a full classifier, LDA().fit(X, y).predict(X), no logistic regression needed — Section 6 runs it, and it is the ancestor that logistic regression (Chapter 11) was invented to compete with: same linear boundaries, fitted by modelling the classes (generative) rather than the boundary directly (discriminative).

📝

Note — two meanings of "explained variance": LDA's explained_variance_ratio_ is the share of between-class variance on each axis — a different denominator from PCA's share of total variance. LD1's 73.9% (next section) and PC1's 36.9% are both correct; they are answering different questions. Comparing them as percentages is a classic misreading.

📋 Stable content — Reviewed: October 2026

5. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Wine.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
lda = LDA(n_components=2)
X_train = lda.fit_transform(X_train, y_train)   # <-- the labels enter here
X_test = lda.transform(X_test)                  # transform never needs them
print(lda.explained_variance_ratio_)            # [0.739  0.261]

from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state=0)
classifier.fit(X_train, y_train)

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
print(confusion_matrix(y_test, y_pred))         # [[14  0  0]
                                                #  [ 0 16  0]
                                                #  [ 0  0  6]]
print(accuracy_score(y_test, y_pred))           # 1.0

Listing 29.2 — The complete supervised pipeline (source: linear_discriminant_analysis.py).

# LDA as a classifier in its own right — one object, no logistic step
clf = LDA().fit(X_train_scaled, y_train)
clf.predict(X_test_scaled)              # Bayes rule with shared covariance
clf.predict_proba(X_test_scaled)        # genuine class probabilities

# the fitted axes, for interpretation
lda.scalings_                           # LD loadings, shape (13, 2)

# small-sample safety: regularise the within-class scatter
LDA(solver='lsqr', shrinkage='auto')    # shrinks S_w toward a scaled identity —
                                        # stabilises it when features ~ samples

Listing 29.3 — LDA's classifier mode, loadings, and shrinkage.

⚠

Warning — supervised leakage, sharpened: PCA's fit had to avoid test features; LDA's fit must not see test labels either — and since fit_transform takes y, the temptation to run it on the whole dataset is one careless line away. Fit on the training set, then bare transform the test set, exactly as listed.

6. The Real Run — Wine, Aimed at the Labels

Fitting LDA on the 142 labelled, scaled training wines returns two discriminants — and this time the eigenvalue ledger reads LD1 73.9% + LD2 26.1% of between-class variance. Compare Chapter 28: PCA's first two axes needed 56.2% of total variance to (almost) do the job; LDA concentrates nearly three-quarters of the class-relevant structure onto a single axis. Aim beats acreage.

Figure 29.1 — All 142 training wines at their real LD coordinates. Cultivar 1 (red) holds the left, cultivar 3 (blue) the right, cultivar 2 (green) spreads through the middle-low. × marks class means: (−4.08, 1.61), (−0.06, −2.49), (4.45, 1.54).

The real geometry, axis by axis:

Class	LD1 range	LD2 range	LD mean
Cultivar 1	−6.25 … −1.61	−0.07 … 3.83	(−4.08, 1.61)
Cultivar 2	−2.45 … 2.56	−5.77 … 0.18	(−0.06, −2.49)
Cultivar 3	2.58 … 5.76	−0.22 … 3.55	(4.45, 1.54)

LD1 is nearly a cultivar ruler: class 1 entirely below −1.6, class 3 entirely above 2.6, no overlap between them on that one number. Class 2 straddles the middle on LD1 — but it owns LD2's low ground (max 0.18, while classes 1 and 3 sit at 1.5+ on average). Two axes, three near-disjoint territories: the labels have been engineered into the geometry.

6.1 The scoreboard

Model (same split, same scaling)	Confusion matrix	Accuracy
LDA-2 + logistic regression	[[14, 0, 0], [0, 16, 0], [0, 0, 6]]	1.000 — perfect
LDA as its own classifier (Bayes rule)	[[14, 0, 0], [1, 15, 0], [0, 0, 6]]	0.972
PCA-2 + logistic (Chapter 28)	[[14, 0, 0], [1, 15, 0], [0, 0, 6]]	0.972
All 13 features + logistic (Ch 28)	[[14, 0, 0], [0, 16, 0], [0, 0, 6]]	1.000
LD1 only + logistic	—	0.833

The headline: two discriminants, a perfect 36 of 36 — matching the full 13-feature model, and edging both the LDA-as-classifier (0.972) and Chapter 28's PCA-2 (0.972). Two readings keep it honest. First, the perfect score's margin is one wine — 36 test samples means a single misclassification costs 2.8 points, and several of these models differ by exactly that one wine (a class-2 wine read as class 1, the same confusable wine PCA missed). Second, the LD1-only row (0.833) is the C − 1 lesson in numbers: with two classes one discriminant would suffice; with three, the middle class needs the second axis.

6.2 What the axes mean — loadings, discounted by noise

LD1 loadings	value	LD2 loadings	value
Flavanoids	−2.201	Proline	+0.913
OD280	−1.018	Alcohol	+0.618
Proline	−0.905	Ash	+0.605
Total phenols	+0.634	Colour intensity	+0.602
Colour intensity	+0.626	Alkalinity of ash	−0.521
Alkalinity of ash	+0.500	Malic acid	+0.421

The chemistry agrees with Chapter 28 — LD1 is dominated by the phenolic cluster (flavanoids at −2.20, OD280, proline), LD2 by the body/alcohol cluster — but notice the magnitudes: Flavanoids' loading is −2.20 where PCA's PC1 loading was +0.43. LDA divides each feature's contribution by its within-class noise, so a feature that separates cultivars sharply relative to its own wobble gets amplified far beyond its raw-variance weight. Same axis, sharpened focus. And the first test wine — the one Chapter 28 followed — lands at LD coordinates (−3.84, 1.55): deep in cultivar-1 territory, classified correctly by every model above.

🌎

Real World: LDA is a workhorse wherever labelled, roughly-Gaussian, moderate-dimensional data appears — fraud scoring, credit rating, biomedical diagnostics, face recognition's classic "Fisherfaces". Its gravity in practice: a century-old method (Fisher, 1936) still in production because its assumptions are checkable, its output is two plottable axes, and it is nearly free to fit.

7. PCA vs LDA — the Honest Ledger

Two chapters, one eigendecomposition, different matrices. The complete comparison:

	PCA (Ch 28)	LDA (this chapter)
Uses labels?	never	yes — fit_transform takes y
Optimises	total variance / min reconstruction error	between-class separation ÷ within-class noise
Eigendecomposes	covariance of X	SḦ⁻¹Sఖ
Max components	min(n−1, features) = 13	classes − 1 = 2
Wine, 2 axes	0.972	1.000
When it wins	no labels; exploratory; compress for storage/denoising; class info hides in low variance	labels exist and are trustworthy; goal is separation; few classes
Fails when	leading variance is class-irrelevant	nonlinear boundaries; unequal class covariances; many classes with few samples each; labels noisy

One subtle trap deserves its own paragraph: supervision can overfit. LDA estimates class means and a shared covariance from the training set; with many features and few samples per class, it will happily find directions that separate the training labels through noise — Chapter 6's bias-variance story wearing eigenvalues. The 178-wine dataset is comfortably in the safe zone (142 samples, 3 classes); a 30-sample, 500-feature gene expression study is not — there, shrinkage (Listing 29.3) or PCA-first (reduce unsupervised, then discriminate) is standard.

💡

Pro Tip — they are not rivals: the professional pipeline for high-dimensional labelled data is often both — PCA to a few dozen components (killing noise and collinearity), then LDA on the scores (aiming at the classes). PCA denoises; LDA aims.

⚠ Volatile content — Reviewed: October 2026 · Next review: January 2027

8. Limitations & the Road Ahead

Where LDA's assumptions bend, and what the family grew into — the method landscape here is the volatile part:

Equal-covariance assumption: when classes have genuinely different spreads, the optimal boundary is quadratic — that is QDA, LDA's sibling: one covariance matrix per class, curved boundaries, more parameters to estimate, needs more data per class.
Linearity: clouds no straight axis can slice (Chapter 14's moons) defeat both PCA and LDA. Next chapter's Kernel PCA applies the kernel trick to dimensionality reduction itself — project into a high-dimensional feature space, do PCA there, and inherit curved coordinates.
Small-n, large-p: with features ≳ samples, SḦ is singular or ill-conditioned. Remedies in daily use: shrinkage regularisation (sklearn's shrinkage='auto'), PCA-first pipelines, or penalised variants from the modern high-dimensional statistics literature.
The discriminant family today: logistic regression (Ch 11) remains the discriminative rival — fewer distributional assumptions, naturally calibrated; regularised LDA holds its own in benchmark studies on well-behaved tabular data; and naive Bayes (Ch 15) is the corner where LDA's covariance is forced to be diagonal.

The parting irony of Part IX so far: PCA followed the data, LDA followed the labels, and both delivered two axes on which three Italian grapes part ways. Chapter 30 drops the last assumption — straightness — and lets the axes themselves bend.

🛠 Hands-On: Aim the Axes YourselfIntermediate

Rebuild both reductions end to end, prove the hand arithmetic, then answer the manager's question: "we can afford to store two numbers per wine — which two, and why not the obvious ones?"

Steps

Hand-verify Section 3 fully: both class means, SḦ = [[16, 0], [0, 4]], w ∝ (1, 2), all four J values (1.0, 1.0, 1.8, 2.0), and the eight projected scores.
Confirm against sklearn on the same eight points: the fitted direction's y/x ratio must be 2.0; project and compare scores.
Hand-check the projection story: along raw x the classes touch at 4; along (1, 2)/√5 the bands are disjoint (A ≤ 3.58 < 5.37 ≤ B).
Run the Wine pipeline (Listing 29.2); verify evr [73.9%, 26.1%], the perfect confusion matrix, and the LD1 class ranges of Section 6.
Run LDA as its own classifier; compare with LDA-2 + logistic and with Chapter 28's PCA-2. Identify the single confusable wine behind every 0.972.
Try n_components=1 (0.833 expected) and n_components=3 — explain the error sklearn raises, from the C − 1 rule.
Bonus: fit PCA and LDA on the same training data and compute the angle between PC1 and LD1 (hint: neither is zero — the data's variance axis and separation axis are related but not identical).

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score

# ---- steps 1-2: the hand example vs the library
Xh = np.array([[0,2],[4,2],[2,1],[2,3],
               [4,4],[8,4],[6,3],[6,5]], float)
yh = np.array([0,0,0,0,1,1,1,1])
mA, mB = Xh[:4].mean(0), Xh[4:].mean(0)
Sw = (Xh[:4]-mA).T @ (Xh[:4]-mA) + (Xh[4:]-mB).T @ (Xh[4:]-mB)
w = np.linalg.inv(Sw) @ (mB - mA)
print(Sw)                       # [[16. 0.] [0. 4.]]
print(w / w[1])                 # [0.5 1. ]  -> direction (1, 2)
for wt in [(1,0),(0,1),(1,1),(1,2)]:
    v = np.array(wt, float)
    print(wt, (v@(mB-mA))**2 / (v@Sw@v))     # 1.0 1.0 1.8 2.0
lda_h = LDA(n_components=1).fit(Xh, yh)
print(lda_h.scalings_.ravel()[1] / lda_h.scalings_.ravel()[0])  # 2.0

# ---- steps 4-6: the Wine pipeline
ds = pd.read_csv('Wine.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.2, random_state=0)
sc = StandardScaler(); Xs_tr = sc.fit_transform(Xtr); Xs_te = sc.transform(Xte)
lda2 = LDA(n_components=2)
T_tr = lda2.fit_transform(Xs_tr, ytr)
print(lda2.explained_variance_ratio_)        # [0.739 0.261]
acc2 = LogisticRegression(random_state=0).fit(
    T_tr, ytr).score(lda2.transform(Xs_te), yte)
print(acc2)                                  # 1.0
clf = LDA().fit(Xs_tr, ytr)
print(confusion_matrix(yte, clf.predict(Xs_te)),
      accuracy_score(yte, clf.predict(Xs_te)))   # 0.972
for cls in [1,2,3]:
    d = T_tr[ytr==cls][:,0]
    print('class %d LD1: %.2f..%.2f' % (cls, d.min(), d.max()))
try:
    LDA(n_components=3).fit(Xs_tr, ytr)
except Exception as e:
    print('n=3 error:', e)

# ---- step 7: angle between PC1 and LD1
from sklearn.decomposition import PCA
pc1 = PCA(1).fit(Xs_tr).components_[0]
ld1 = lda2.scalings_[:,0]
cosang = abs(pc1 @ ld1) / (np.linalg.norm(pc1)*np.linalg.norm(ld1))
print('angle PC1 vs LD1: %.1f degrees' % np.degrees(np.arccos(cosang)))

Expected results (real): SḦ [[16, 0], [0, 4]]; direction ratios 0.5/1.0 (sklearn agrees: 2.0); J = 1.0, 1.0, 1.8, 2.0. Wine: evr [0.739, 0.261]; LDA-2 + logistic 1.000 (cm [[14,0,0],[0,16,0],[0,0,6]]); LDA-as-classifier 0.972 (cm [[14,0,0],[1,15,0],[0,0,6]] — the same single class-2→1 wine PCA-2 missed); LD1-only 0.833; n_components=3 raises a warning/error because only 2 discriminants exist (C − 1 = 2). The PC1–LD1 angle is small — the phenolic axis serves both variance and separation here — but not zero: the two questions are related, not identical.

🎯 Key Takeaways

LDA is PCA with one word changed: the axes are fitted with the labels. Same eigen-machinery, matrix swapped — SḦ⁻¹Sఖ instead of plain covariance.

Fisher's ratio is separation over noise: hand-verified w ∝ SḦ⁻¹(gap) = (1, 2) — the 4-unit x-gap discounted 16-fold loses to the 2-unit y-gap discounted 4-fold.

J((1,2)) = 64/32 = 2.0 beats (1,0), (0,1) and (1,1); on the winning axis the eight hand points fall into disjoint bands — noise-aware aim beats naive gap-chasing.

Components cap at classes − 1: three cultivars, two discriminants. LD1 alone orders cultivars 1 < 2 < 3 but only scores 0.833 — the middle class needs LD2.

Real run: LD1 carries 73.9% of between-class variance; LDA-2 + logistic scores a perfect 1.000 (36/36) vs PCA-2's 0.972 and 13 features' 1.000 — aim beats acreage.

Supervision is a loan: Gaussian classes, equal covariance, labels at fit time, overfitting risk when samples are scarce. No labels or exploratory work — PCA is the honest choice.