Part II — Regression

Chapter 9 · Random Forest Regression

One tree memorises and wobbles. A forest averages many wobbling trees into a steady prediction — the closest machine learning gets to a statistical free lunch. This chapter makes the lunch honest: bootstrap sampling simulated by hand, the variance mathematics of averaging, the smoother staircase on the salary grid, the n_estimators dial with real numbers, and the out-of-bag score that validates a forest for free.

📖 Chapter 9 of 33 ⏱ ~85 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Explain why averaging many unstable trees produces a stable model — and prove it with the variance formula
Simulate bootstrap sampling by hand and derive the 63.2% rule
Describe both sources of randomness in a random forest and the role each plays in de-correlation
Read the forest's smoother staircase on Position_Salaries.csv, with real numbers against the single tree
Use n_estimators correctly — what it does, what it costs, and what it cannot fix
Explain out-of-bag scoring as free validation, and interpret the real OOB result on this dataset

1. The Wisdom of Averaged Trees

Chapter 8 ended on a diagnosis: a single tree is a high-variance learner. Resample the data and the greedily chosen splits rearrange — different walls, different predictions, same data-generating world. Chapter 2 planted the cure centuries early: the variance of an average is smaller than the variance of its parts. A random forest (Breiman, 2001) is exactly that cure, industrialised:

grow B trees on B different random slices of the data, predict with the average of their predictions

Each individual tree remains as unstable as ever — forests do not fix trees, they outvote them. The ensemble's error is not any tree's error; it is the error of the crowd, and crowds of diverse, competent members beat their members. Two conditions make the crowd work, and they structure this chapter: diversity (trees must disagree — Sections 2–3) and individual competence (each tree must be better than chance — the per-tree dials of Chapter 8 still apply).

🌎

Real World: Ask one analyst to estimate a salary and you get their biases; ask a hundred analysts with different data and different habits, then average — the idiosyncrasies cancel, the shared signal survives. Galton weighed an ox with a crowd in 1906; random forests weigh salaries the same way. The modern tabular-data leaderboard (Chapter 32's boosted trees aside) is still dominated by this idea's descendants.

2. Bootstrap & Bagging

Where do B different datasets come from, when you only have one? The bootstrap answer: manufacture them by sampling your own rows with replacement. Draw n rows from your n rows, putting each back after drawing — some rows appear twice or thrice, others vanish. Repeat per tree.

2.1 The 63.2% rule — verified by hand

How many distinct rows survive one bootstrap draw of n = 10? Each row dodges all ten draws with probability (9/10)¹⁰ ≈ 0.349 — so about 34.9% of rows vanish and ≈ 65.1% ≈ 63.2% survive (the classic limit for large n: 1 − 1/e). We simulated it for real: 10,000 bootstrap draws of ten rows averaged 6.50 unique rows per tree — theory, meet arithmetic.

Figure 9.1 — Bagging: bootstrap replicates feed one tree each; the forest averages.

Bagging — bootstrap aggregating — is the name of this whole recipe. Its statistical payload is one line: if B estimators each have variance σ² and are independent, their average has variance σ²/B. Averaging a hundred equally-noisy predictors divides the wobble by a hundred.

⚠

Warning — the catch: trees grown on overlapping bootstraps of the same data are anything but independent; they share the strongest splits (every tree will find the CEO wall). For correlated estimators the variance of the average floors out at ρσ² — averaging beyond that ceiling buys nothing. The forest's second trick exists to push ρ down; that is Section 3.

3. Random = De-correlated

The variance of a forest of B trees, each with variance σ² and pairwise correlation ρ, is exactly:

Var(forest) = ρσ² + (1 − ρ)σ² / B

The second term melts away as B grows; the first — the shared wobble — remains. Shrinking ρ attacks the floor itself. Bagging lowers ρ a little (different data slices); feature subsampling lowers it more: at every split, the tree considers only a random subset of features (max_features), so no single dominant feature dictates every tree's structure. Different trees grow genuinely different opinions — which is precisely what makes their average smart.

An honest note for this chapter's dataset: with exactly one feature, feature subsampling is idle (sklearn's regression default max_features=1.0 considers the one feature everywhere). The forest here is random only through its bootstraps — and even those are gentle on ten tidy rows. The machinery matters at scale: with 40 features, max_features is what keeps the crowd diverse. Small dataset, big lesson: know which randomness is actually working for you.

💡

Pro Tip: The two randomnesses have two purposes. Bootstrap randomness makes each tree's data differ; feature randomness makes each tree's perspective differ. Remove either and ρ climbs — slowly with the first, sharply with the second on wide data.

4. The Smoother Staircase

What does averaging ten staircases look like? Each tree still predicts flat steps, but their walls stand at different places (each tree saw ~6.5 of the 10 rows). The average of many offset staircases is a finer staircase — same shape family, gentler jumps, noise-free steps blended:

Figure 9.2 — Averaging offset staircases: the forest (solid) tracks the same data with finer, gentler steps than any single tree (dashed).

The real numbers against Chapter 8's tree:

Model	R² (in-sample)	Pred @6.5	Pred @12
Single tree (Ch 8)	1.0000	₹150,000	₹1,000,000
Forest, 10 trees (source)	0.9704	₹167,000	₹850,000

Read that carefully, because it looks like a demotion and is actually the point. The single tree's R² = 1.0 is memorisation — ten singleton leaves. The forest scores 0.97 because averaging diluted the memorisation: each tree fitted only its ~6.5 rows' quirks, and quirks do not survive averaging. On clean data the dilution costs a little in-sample glitter; on noisy data it is exactly what buys generalisation. And note the @12 extrapolation: ₹850,000 — still flat (a forest cannot leave the range any more than its trees can), but the CEO's million is now averaged against trees that never saw the CEO row — even our extrapolations have error bars in spirit.

5. n_estimators: the Free-ish Dial

How many trees? Unlike every previous dial, this one carries no overfitting risk: more trees only smooth the average. But the prediction itself is noisy when B is small — which trees happened to be drawn — and stabilises as B grows. The real sweep (same data, same seed):

n_estimators	1	2	5	10	50	100	300
R² (in-sample)	0.946	0.984	0.940	0.970	0.957	0.938	0.953
Pred @6.5	₹200,000	₹155,000	₹152,000	₹167,000	₹161,200	₹158,300	₹160,333

Two real lessons in that table. First, look at the @6.5 row with small B: ₹200,000 with one tree, ₹155,000 with two — the prediction itself is a random variable until enough trees average out. Second, watch it settle: by B = 300 the answer stabilises at ₹160,333 — which is, almost exactly, the mean of the two salaries bracketing level 6.5 (₹150k and ₹170k average to ₹160k): the crowd's considered opinion lands within ₹333 of the candidate's claim. The R² wobble across B (0.94–0.98) is the same stabilisation seen in-sample.

💡

Pro Tip: n_estimators trades compute for stability, not fit — doubling trees never "overfits harder", it just costs CPU. Practical default: 100 (sklearn's), raised when predictions must be reproducible to the rupee or the variance formula's second term still shows. The dial that does control fit lives inside each tree — max_depth, min_samples_leaf — Chapter 8's dials, now per-tree.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

Same skeleton as the tree — the ensemble hides inside one estimator object:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:-1].values
y = dataset.iloc[:, -1].values

from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor(n_estimators=10, random_state=0)
regressor.fit(X, y)

print('R2   :', regressor.score(X, y))          # 0.9704
print('@6.5 :', regressor.predict([[6.5]]))     # [167000.]
print('trees:', regressor.n_estimators)

# Smoother staircase on the fine grid
X_grid = np.arange(min(X), max(X), 0.01).reshape(-1, 1)
plt.scatter(X, y, color='red')
plt.plot(X_grid, regressor.predict(X_grid), color='blue')
plt.title('Truth or Bluff (Random Forest Regression)')
plt.xlabel('Position level'); plt.ylabel('Salary')
plt.show()

Listing 9.1 — The source forest: ten trees, one line of extra code (source: random_forest_regression.py).

Two production-grade extensions, both one parameter away:

# More trees: stable answers
forest = RandomForestRegressor(n_estimators=300, random_state=0).fit(X, y)
print(forest.predict([[6.5]]))                   # [160333.33...]

# Out-of-bag scoring: validation built into training (Section 7)
forest = RandomForestRegressor(n_estimators=100, random_state=0,
                               oob_score=True).fit(X, y)
print('OOB R2:', forest.oob_score_)             # 0.6286

Listing 9.2 — Convergence and the free validation score.

💡

Pro Tip: Because trees are independent, forest training parallelises perfectly — set n_jobs=-1 to use every core. The forest is one of the few models that gets faster on bigger machines for free.

7. OOB — the Free Validation

Every bootstrap leaves ≈ 36.8% of rows out of that tree's sample (Section 2's arithmetic, flipped). The forest exploits this elegantly: to score row i, let only the trees that never saw row i vote on it. Every row is thus predicted by a sub-forest that was blind to it — an honest, holdout-style estimate computed from the training run itself, at zero extra cost:

OOB score = R² computed on every row, each scored only by its blind trees

Real result on this data (100 trees): OOB R² = 0.6286 against in-sample 0.9378. The gap is the memorisation tax made visible — the same dilution Section 4 discussed, now with a number attached. OOB is a preview of Chapter 31's cross-validation, obtained without a single extra fit.

Treat the OOB number as you would any honest estimate: on ten rows it is coarse (each row's blind sub-forest is a small crowd), but on thousands of rows OOB tracks k-fold cross-validation closely and is the practitioner's first glance at "am I fooling myself?" — before paying for the full validation machinery.

⚠

Warning: OOB honesty applies to the rows, not to your tuning loop. If you compare fifty forest configurations by OOB and keep the best, the winning OOB score is itself an optimised statistic — the multiple-comparisons trap Chapter 5 flagged for p-values, reincarnated. Chapter 31 closes that loop properly.

8. Part II Retrospective & Limits

Six regressors now share your workbench, and they have all answered the same negotiation question. The complete scoreboard, every number real:

Model (Ch)	Key dial	Fit (in-sample R²)	@6.5	@12	Character
Linear (4)	—	0.669	₹330,379	₹775,212	Transparent, straight
Poly deg-4 (6)	degree	0.997	₹158,862	₹3,035,985	Bends; detonates outside
SVR tuned (7)	C, ε, γ	0.989	₹163,534	₹1,174,312	Tube + bumps
Tree (8)	max_depth	1.000	₹150,000	₹1,000,000	Auditable; memorises
Forest (9)	n_estimators + per-tree	0.970	₹160,333	₹850,000	Stable crowd

The forest's own verdict on the negotiation — ₹160,333 against a claim of ₹160,000 — is the closest of the family, and for the least effort. Its limitations carry over from its members: still step-shaped (no slopes), still flat beyond the range, and now also opaquer than one tree (audit a hundred trees? only in aggregate). Its statistical personality is the mirror of the single tree's: low variance, similar bias — and where variance is the enemy, forests are the default tool of choice.

Which raises the question the scoreboard makes unavoidable: in-sample R² crowned the memorising tree — how should we actually rank these models? That is precisely Chapter 10: honest metrics (MAE, RMSE, residual audits), held-out evaluation, and a formal benchmark of all six regressors under one system. The algorithms are assembled; the judgement layer is next.

🛠 Hands-On: Assemble the CrowdBeginner

Management liked the single tree's auditability but not its mood swings. Deliver the forest: source configuration, a convergence check, the free OOB score, and the final negotiation answer with a stability argument.

Steps

Fit the source forest (10 trees, random_state=0); report R² and @6.5.
Sweep n_estimators over {1, 2, 5, 10, 50, 100, 300}; tabulate @6.5. At which B does the answer stabilise, and to what value?
Fit with oob_score=True (100 trees). Compare OOB with in-sample R² and explain the gap in one sentence.
Simulate 10,000 bootstrap draws of 10 rows; verify the mean unique-row count against the 63.2% rule.
Compare forest vs single tree on @6.5 and @12. Which limitations survived the averaging?
Write the negotiation verdict: model, answer, stability evidence, and the standing extrapolation ban.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.ensemble import RandomForestRegressor

ds = pd.read_csv('Position_Salaries.csv')
X = ds.iloc[:, 1:-1].values.astype(float)
y = ds.iloc[:, -1].values.astype(float)

r = RandomForestRegressor(n_estimators=10, random_state=0).fit(X, y)
print('source forest: R2=%.4f  @6.5=%.0f' % (r.score(X, y), r.predict([[6.5]])[0]))

for n in [1, 2, 5, 10, 50, 100, 300]:
    f = RandomForestRegressor(n_estimators=n, random_state=0).fit(X, y)
    print('B=%3d  @6.5=%9.0f  R2=%.4f' % (n, f.predict([[6.5]])[0], f.score(X, y)))

f = RandomForestRegressor(n_estimators=100, random_state=0, oob_score=True).fit(X, y)
print('OOB=%.4f  in-sample=%.4f' % (f.oob_score_, f.score(X, y)))

rng = np.random.RandomState(0)
print('mean unique rows:', np.mean(
    [len(set(rng.randint(0, 10, 10))) for _ in range(10000)]))

Expected results (real):

Step 1: R² = 0.9704, @6.5 = ₹167,000.
Step 2: @6.5 runs 200,000 / 155,000 / 152,000 / 167,000 / 161,200 / 158,300 / 160,333 — stable by B ≈ 100, settling at essentially (150k + 170k)/2: the two bracketing salaries.
Step 3: OOB = 0.6286 vs in-sample 0.9378 — the gap is the memorisation tax: blind trees can't recall rows they never saw.
Step 4: ≈ 6.50 unique rows (theory 6.32; small-n noise).
Step 5: @6.5: forest ₹160,333 vs tree ₹150,000 (forest's averaged answer is stabler); @12: ₹850,000 vs ₹1,000,000 — still flat, gently hedged. Survived limitations: no slopes, no extrapolation, opacity.
Step 6: "Random forest (300 trees): ₹160,333 — within 0.2% of the claim; prediction stable within ₹2k from B=100; OOB confirms honest fit on blind trees; answers valid for levels 1–10 only."

🎯 Key Takeaways

Forests don't fix unstable trees — they outvote them: average many diverse, competent learners and idiosyncrasy cancels while signal survives.

Bootstrap = sample rows with replacement: ~63.2% unique rows per tree (verified: 6.50 of 10). Bagging = bootstrap + aggregate.

Var(forest) = ρσ² + (1−ρ)σ²/B. Averaging melts the second term; de-correlation (bootstrap + feature subsampling) attacks the ρσ² floor.

The forest's staircase is finer than any tree's; its R² 0.97 below the tree's 1.0 is dilution of memorisation — the healthy kind of worse.

n_estimators buys stability, not fit: @6.5 swung ₹152k–₹200k at small B and settled at ₹160,333 by B = 300. Cost is CPU only.

OOB scores each row with its blind trees — free holdout validation (real: 0.63 vs 0.94). Steps, flatness and opacity survive the averaging.