Part III — Classification

Chapter 17 · Random Forest Classification & Ensemble Learning

One tree is a child of its sample; a forest is a committee of them. Chapter 9's averaging cure returns in classification dress — bootstrap, subsample, vote — with three new payoffs: graded probabilities from vote shares, feature importances for free, and a stability experiment that measures variance cancellation directly. Then the definitive bagging-vs-boosting table, closing the ensemble story Chapter 32 will finish.

📖 Chapter 17 of 33 ⏱ ~85 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Apply Chapter 9's bagging machinery to classification: bootstraps, votes, averaged vote shares
Explain feature subsampling's de-correlation role — and what sqrt(p) means for a two-feature dataset
Read the real forest: confusion matrix, n_estimators sweep, OOB score, and the near-tied feature importances
Interpret the seed-stability experiment: more trees, fewer prediction flips
State the bagging-vs-boosting distinction precisely, ready for Chapter 32
Place the forest honestly on the eight-model scoreboard — including where a single pruned tree beats it

1. The Instability Bill, Due

Chapter 16 closed on a warning you have heard before: the fitted tree is a child of its sample. Resample the 300 rows and the thresholds move — 44.5 becomes 42 or 47, maybe salary leads this time. In Chapter 6's language: a single deep tree is a high-variance learner. Chapter 9 already wrote the cure for regression — average many unstable learners into a stable one — and nothing about it was regression-specific. This chapter cashes it for classification, where the average becomes a vote and the payoff includes something regression never offered: graded probabilities without a single sigmoid in sight.

📝

Note — same dataset, same discipline: 400 users, the Chapter 11 split (300/100, seed 0), scaled features (harmless habit — verified again: raw and scaled forests differ by one test row, 0.92 vs 0.91, bootstrap tie-break wobble, not unit sensitivity). The source's configuration: n_estimators=10, criterion='entropy'.

2. Bagging Votes

The recipe, unchanged from Chapter 9 until the last step:

Bootstrap: draw n rows with replacement from the training set — about 63.2% distinct rows per tree, the rest repeats.
Grow one (deep, unpruned) tree per bootstrap.
Vote: a new customer falls into one leaf per tree; each tree answers with its leaf's majority class; the forest predicts the majority of majorities.

Figure 17.1 — Bagging in classification: B bootstrap trees, one vote each, majority rules — and the vote share doubles as a probability.

2.1 The graded-probability bonus

Chapter 16's single tree reported probabilities in pure-leaf steps — [0.0, 1.0] or [1.0, 0.0], unanimous and unnuanced. The forest's predict_proba is the average vote share across trees, and averaging smooths the steps into a genuine grade. Real outputs for our anchor customer (30, ₹87k):

Model	P(no) , P(yes)	Personality
Single tree (Ch 16)	1.0 , 0.0	One pure leaf's false certainty
Forest, 100 trees	0.90 , 0.10	90 of 100 trees said no — strong but graded
Naive Bayes (Ch 15)	0.899 , 0.101	Posterior, for comparison — near-identical verdict here

That convergence — a vote of 100 trees landing on Naive Bayes's posterior to three decimals — is coincidence, not law, but the shape of the lesson holds: averaging converts a tree's brittle 0-or-1 into a defensible confidence level. For the fence-sitting customer (45, ₹87k), the forest's [0.45, 0.55] is arguably the most honest statement any model has made about them.

3. The "Random": Feature Subsampling

Bootstraps alone make trees differ, but Chapter 9's variance formula named the real enemy: correlation ρ. All bootstrap trees still see the same features, so they keep finding the same strong splits (everyone leads with age) and their votes stay correlated — the ρσ² floor. The second randomness attacks ρ directly: at every split, each tree considers only a random subset of features (max_features):

classification default: max_features = √p · regression default (Ch 9): p

With p features, each split consults a random √p of them — no single dominant feature can dictate every tree's structure. Diversity of perspective, by construction.

On this dataset, a lovely edge case: √2 ≈ 1, so every split in every tree consults exactly one randomly chosen feature — age or salary, coin-flip per node. Maximum possible de-correlation: half the trees' splits can't even see what the other half saw. Small data, extreme version of the general principle; on a 100-feature problem, √100 = 10 features per split keeps the crowd diverse without starving it.

💡

Pro Tip — why classification uses fewer: classification trees vote, and diverse wrong answers cancel; regression trees average numbers, where richer per-tree information often helps. Hence √p versus p — a defensible default, tunable like everything else (Chapter 31).

4. The Real Forest

The source configuration (10 trees, entropy) on the standard split:

Metric	Value (real)
Confusion matrix	[[63, 5], [4, 28]] — 5 wasted ads, 4 missed buyers
Test accuracy	0.91
Feature importances	Age 0.498 · Salary 0.502 — a near-tie

4.1 The n_estimators sweep — memorisers that vote well

Trees	1	5	10 (source)	50	100	300
Test acc	0.88	0.91	0.91	0.92	0.91	0.92
Train acc	0.933	0.963	0.980	1.000	1.000	1.000

The paradox worth savouring: from 50 trees on, every individual tree memorises its bootstrap (train accuracy 1.000 — they are unpruned, after all), yet the committee's test score holds at 0.91–0.92. Each tree's quirky memorised errors live in different rows; the vote cancels what the individuals share no version of. Variance cancellation is not a metaphor — it survives unanimous memorisation.

4.2 OOB — the free referee

With 100 trees: OOB score 0.87 against in-sample 1.000. Each row graded only by the ~37% of trees that never saw it — the honest estimate sits three points under the committee's glory, and one under its final test score. Chapter 9's caveats apply word for word, including: don't shop for hyperparameters by repeatedly consulting OOB.

4.3 The stability experiment — variance cancellation, measured

Chapter 16's instability was qualitative ("resample and the tree changes"). Here is the quantitative version: we trained forests with five different seeds and counted how many of the 100 test predictions changed:

Forest size	Predictions differing vs seed 0 (of 100)
10 trees	7, 6, 4, 5 — half a dozen flips per reseed
100 trees	1, 1, 3, 1 — near-frozen

Ten trees still wobble — a small committee is hostage to which bootstraps happened to be drawn. One hundred trees barely move at all. That is ρσ² + (1−ρ)σ²/B doing exactly what the formula promised, measured where it matters: in predictions.

⚠

Warning — importances ≠ root position: the single tree led with age (root: Age ≤ 44.5), yet the forest scores the features a dead heat (0.498 vs 0.502). Importance aggregates each feature's contribution across every split in every tree — salary, never the root on the full data, accumulates nearly half the total through hundreds of subordinate cuts. Use importances to rank features; never to crown "the most important split."

📋 Stable content — Reviewed: August 2026

5. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators=10, criterion='entropy',
                                    random_state=0)
classifier.fit(X_train, y_train)

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
print(confusion_matrix(y_test, y_pred))   # [[63  5]
                                          #  [ 4 28]]
print(accuracy_score(y_test, y_pred))     # 0.91

# The extras every forest ships with
print(classifier.feature_importances_)    # [0.498  0.502]
print(classifier.predict_proba(
    sc.transform([[45, 87000]])))         # [[0.45  0.55]] — honest doubt

Listing 17.1 — The source forest and its two free reports (source: random_forest_classification.py).

# Stability, measured — the Section 4.3 experiment
for n in [10, 100]:
    flips = []
    for seed in range(1, 5):
        f1 = RandomForestClassifier(n_estimators=n, random_state=0).fit(X_train, y_train)
        f2 = RandomForestClassifier(n_estimators=n, random_state=seed).fit(X_train, y_train)
        flips.append(int((f1.predict(X_test) != f2.predict(X_test)).sum()))
    print(f'{n} trees: {flips} predictions flip per reseed')

# OOB, one parameter away
f = RandomForestClassifier(n_estimators=100, oob_score=True,
                           random_state=0).fit(X_train, y_train)
print(f.oob_score_)                       # 0.87

Listing 17.2 — Reproducing the stability experiment and the free validation score.

💡

Pro Tip — the production defaults: n_jobs=-1 (trees train independently — every core for free) and 100+ trees for prediction stability, per the experiment above. The 10 in the source is a teaching configuration, not a recommendation.

6. Bagging vs Boosting — the Definitive Table

Two families make an ensemble, and they could not disagree more about how. Bagging (this chapter) builds independent learners in parallel and averages; boosting (Chapter 32's XGBoost) builds learners in sequence, each one focused on its predecessors' mistakes:

	Bagging (forest)	Boosting (XGBoost, Ch 32)
Construction	Parallel — independent trees on random bootstraps	Sequential — each tree fits the previous ensemble's errors
Primary target	Variance: cancels individual wobble	Bias: chips away residual errors
Learner style	Deep, low-bias, high-variance trees (memorisers welcome)	Shallow, weak stumps that improve in steps
Combination	Majority vote / mean	Weighted sum, weights learned per round
Overfitting risk	Low — more trees never hurt (real: train 1.000, test holds)	Real — rounds chase the training set; needs early stopping
Sensitivity to noisy labels	Robust (noise is outvoted)	Fragile (noise is chased)
Parallelisable	Perfectly (`n_jobs=-1`)	Inherently sequential per round
When it wins	Big messy data, variance-limited learners	Structured data, accuracy at any cost, tuning budget

Figure 17.2 — The two ensemble philosophies: independent crowd vs sequential chain of corrections.

🌎

Real World: the modern tabular-data leaderboard is owned by gradient-boosted trees — XGBoost, LightGBM, CatBoost — with random forests the strong, low-maintenance runner-up. The practitioner's rule of thumb: prototype with a forest (hard to misuse), compete with boosting (wins when tuned). Chapter 32 shows why the tuning burden exists.

7. Reading the Results: the Honest Scoreboard

	Logistic	K-NN	SVM lin	SVM RBF	NB	DT full	DT d-2	Forest
Accuracy	0.89	0.93	0.90	0.93	0.90	0.91	0.94	0.91–0.92
Errors	11	7	10	7	10	9	6	9
Model size	3 coefs	300 rows	128 SVs	88 SVs	9 stats	50 leaves	4 leaves	10–300 trees
Native output	prob	vote share	distance	distance	posterior	leaf share	leaf share	avg vote share

The honest headline: the forest did not win this dataset — the depth-2 single tree (0.94) and the flexible duo (K-NN, RBF at 0.93) beat it, and the forest merely matches its own unpruned parent. Why, when forests are famous for winning? Because this dataset's true structure is a clean, boxy, three-rule region — exactly the shape one pruned tree captures perfectly — while the forest's deep bootstrap trees vote on fragments of that structure, diluting the crisp rules with democratic noise. Forests earn their reputation on data that is bigger, messier, and less boxy than 400 tidy rows; the scoreboard here is a lesson in matching tools to structure, not a referendum on ensembles.

What the forest did win, even here: confidence quality (graded [0.45, 0.55] on the fence-sitter vs a single tree's fake [0, 1]), stability (1–3 flips per reseed at 100 trees), and freedom from tuning (no depth search needed — defaults matched the tuned single tree within a point).

7.1 The boundary customer, sixth opinion

(30, ₹150k): forest says [0.03, 0.97] — yes, with 97 of 100 trees agreeing. The verdict now reads like a committee report: logistic abstained (0.49), the linear SVM edged yes (+0.21), RBF and NB and the single tree committed firmly, and the forest's 97–3 vote is as close to unanimity as this cast has produced. Six models, one growing consensus — and next chapter finally builds the instrument that should have been in the room all along: an honest, cost-aware way to score them all.

8. Strengths, Limits & Part III's Toolkit

Strengths	Limitations
Variance-cancelling defaults — hard to misuse, no depth tuning required	Opaque: no single readable tree (importances are the compromise)
Graded probabilities from vote shares; free OOB validation	Can lose to one pruned tree on small, clean, boxy data (real: 0.91 vs 0.94)
Feature importances for free; scale-blind; mixed types welcome	Prediction cost = all trees consulted (fast in aggregate, not free)
Perfectly parallel training; strong on big messy tabular data	Still flat beyond the training range (inherited from trees)

Part III's workbench is now full: equations (logistic), neighbourhoods (K-NN), streets straight and curved (SVM, kernel SVM), beliefs (Naive Bayes), rules (tree) and committees (forest). Seven classifiers, seven philosophies, one shared crutch — every one of them has been graded by a single blunt number: accuracy. Chapter 18 replaces that crutch with the full judgement layer — precision, recall, F1, ROC curves and AUC — and finally reads the confusion matrices this Part has been quietly collecting.

🛠 Hands-On: Committee of TreesIntermediate

Convince a sceptic that "averaging unstable trees" does anything. Run the forest, the sweep, OOB, the importances — and the stability experiment that turns the claim into a number.

Steps

Fit the source forest (10 trees, entropy); verify [[63, 5], [4, 28]] and 0.91.
Run the n-sweep {1, 5, 10, 50, 100, 300} with train and test accuracy; explain the train-1.000/test-holds paradox.
Fit with oob_score=True (100 trees); report OOB vs in-sample and interpret the gap.
Report feature importances; contrast with the single tree's age-led root and explain the near-tie.
Run the stability experiment (5 seeds × {10, 100} trees); tabulate prediction flips.
Collect predict_proba for the three anchors; compare confidence quality against the single tree and NB. Write the one-paragraph verdict on what the forest bought.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
Xs_tr, Xs_te = sc.fit_transform(X_tr), sc.transform(X_te)

rf = RandomForestClassifier(n_estimators=10, criterion='entropy',
                            random_state=0).fit(Xs_tr, y_tr)
print(confusion_matrix(y_te, rf.predict(Xs_te)), rf.score(Xs_te, y_te))

for n in [1, 5, 10, 50, 100, 300]:
    m = RandomForestClassifier(n_estimators=n, criterion='entropy',
                               random_state=0).fit(Xs_tr, y_tr)
    print('n=%3d test=%.3f train=%.3f' % (n, m.score(Xs_te, y_te), m.score(Xs_tr, y_tr)))

f = RandomForestClassifier(n_estimators=100, criterion='entropy',
                           oob_score=True, random_state=0).fit(Xs_tr, y_tr)
print('OOB %.3f  in-sample %.3f  importances %s' %
      (f.oob_score_, f.score(Xs_tr, y_tr), f.feature_importances_.round(3)))

for n in [10, 100]:
    base = RandomForestClassifier(n_estimators=n, random_state=0).fit(Xs_tr, y_tr).predict(Xs_te)
    flips = [int((RandomForestClassifier(n_estimators=n, random_state=s)
                  .fit(Xs_tr, y_tr).predict(Xs_te) != base).sum()) for s in range(1, 5)]
    print(f'{n}-tree forests flip {flips} of 100 predictions per reseed')

for cust in [[30, 87000], [30, 150000], [45, 87000]]:
    print(cust, f.predict_proba(sc.transform([cust]))[0].round(3))

Expected results (real):

Steps 1–2: 0.91 at n=10; sweep 0.88 / 0.91 / 0.91 / 0.92 / 0.91 / 0.92 test with train reaching 1.000 from n=50 — unanimous memorisers, stable committee.
Step 3: OOB 0.87 vs in-sample 1.000 — the free referee prices the optimism at 13 points; test lands between (0.91).
Step 4: [0.498, 0.502] — near-tie despite the age-led single-tree root: importance sums every split in every tree.
Step 5: 10-tree forests flip 4–7 of 100 predictions per reseed; 100-tree forests 1–3 — the formula's promise, measured.
Step 6: [0.90, 0.10] / [0.03, 0.97] / [0.45, 0.55]: graded confidence where the single tree said 1.0/0.0/0.0 — the forest's purchase is honesty about uncertainty, stability, and tuning-free defaults, at the price of readability and one scoreboard point to the pruned tree.

🎯 Key Takeaways

Forests vote: B bootstrap trees, majority rules, and the average vote share doubles as a graded probability — [0.90, 0.10] where one tree said [1, 0].

Feature subsampling fights correlation: √p features per split — here just 1 of 2, maximum diversity. Classification votes; regression averages — hence √p vs p.

From 50 trees on, every tree memorises (train 1.000) yet the committee holds 0.91–0.92 on test — variance cancellation survives unanimous memorisation.

Stability, measured: 10-tree forests flip 4–7 of 100 predictions per reseed; 100-tree forests 1–3. More trees = the same answer, reproducibly.

Importances [0.498, 0.502] ≠ the root: importance aggregates every split in every tree. OOB 0.87 is the free referee against in-sample glory.

Bagging averages independent voters to kill variance; boosting chains correctors to kill bias (Ch 32). The forest's honest loss here (0.91 vs 0.94) is a lesson in matching tool to data shape.