Part II — Regression

Chapter 8 · Decision Tree Regression

No equation, no kernel, no gradient — just questions. A decision tree carves the feature space into axis-aligned boxes and answers with the average of whoever lives inside each box. It is the most interpretable model in this book, the only one that never needs scaling, and — left uncontrolled — a perfect memoriser. Same ten salaries, a model you can literally draw.

📖 Chapter 8 of 33 ⏱ ~80 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Explain how a regression tree learns: greedy, axis-aligned splits that maximise variance reduction
Compute a split's variance reduction by hand, and explain why leaves predict means
Read the step-shaped prediction curve and the tree diagram as two views of one model
State precisely why trees are scale-invariant — and why that makes preprocessing easier than any model so far
Use tree depth as the bias-variance dial, with real numbers from depth 1 to unlimited
Read a fitted tree's actual rules, describe its flat extrapolation behaviour, and name its instability problem — setting up Chapter 9's fix

1. A Model You Can Draw

Part II now owns two philosophies: equations (Chapters 4–6 — coefficients fitted by least squares) and geometry (Chapter 7 — kernel bumps in implicit space). The decision tree takes a third road: logic. The model is a flowchart of yes/no questions — "Is level ≤ 8.5?" — ending in a prediction at each leaf. Nothing is fitted by calculus; the tree is grown, one question at a time, each chosen to make the answers less messy.

The idea traces to CART — Classification And Regression Trees (Breiman et al., 1984) — one algorithm family serving both supervised tasks: classification trees vote (Chapter 16), regression trees average (this chapter). On the salary grid the promise is immediate: after fitting, you can print the entire model and read it to HR — no coefficients to interpret, no kernels to imagine.

📝

Note: Same dataset, same question as Chapters 6–7 — what salary does level 6.5 deserve? By the end of this chapter you will have the tree's answer (₹150,000, as it happens) plus the ability to explain exactly why it says so — a transparency none of the previous models offered.

2. How a Tree Learns: Splits & Variance Reduction

A regression node holds a set of training rows. Growing the tree means finding, among all (feature, threshold) pairs, the split that makes the two children's salaries as internally consistent as possible. Formally, each candidate split is scored by variance reduction:

Δ = Var(parent) − (n_L/n)·Var(left) + (n_R/n)·Var(right)

Var here is the plain variance of the training y's inside the node. The split with the largest Δ wins; the process recurses on each child until a stopping rule fires. A leaf's prediction is the mean of its residents' y — the constant that minimises squared error within the leaf, Chapter 4's logic in miniature.

2.1 Worked by hand — one split, all arithmetic

Four employees, salaries y = {1, 2, 3, 10} (₹ lakh) at levels x = {1, 2, 3, 4}. Candidate split: level ≤ 2.5.

Parent: mean = 4; variance = [(−3)² + (−2)² + (−1)² + 6²] / 4 = (9+4+1+36)/4 = 12.5
Left {1,2,3}: mean 2; variance = (1+0+1)/3 = 0.667
Right {10}: one point; variance = 0
Weighted children: (3/4)(0.667) + (1/4)(0) = 0.5

Δ = 12.5 − 0.5 = 12.0

Try the rival split (level ≤ 1.5): left {1} variance 0, right {2,3,10} variance = (1+0+49)/3 = 16.67, weighted = (1/4)(0) + (3/4)(16.67) = 12.5, Δ = 0 — useless. The tree's greed is arithmetic: isolate the outlier first. That is exactly what the real fit does on the salary grid (Section 7): the root split severs the ₹1,000,000 CEO from everyone else, because that single separation erases the most variance in one move.

💡

Pro Tip: "Greedy" means each split is chosen for maximum immediate Δ with no lookahead — the globally best tree may require a locally worse first split, and greedy search will never find it. Trees trade optimality for speed and interpretability; Chapter 9's forests patch the consequences statistically.

3. The Step-Shaped Prediction Curve

Every split erects a wall: on one side the model predicts one leaf-mean, on the other side another. Prediction is therefore constant within each box — the fitted "curve" is a staircase, not a curve at all:

Figure 8.1 — The full-depth fitted tree on ten salaries: one wall per midpoint, one flat step per leaf. The "curve" is the data, replayed.

The real fitted tree puts walls at every midpoint — 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5 — because with unlimited depth nothing stops it from isolating each employee. Each leaf holds exactly one salary, so each step is an observed data point. The prediction for any new level is simply the salary of the nearest employee on the appropriate side of a wall: level 6.5 falls at the wall itself, goes left (≤ 6.5), and inherits the level-6 leaf: ₹150,000.

⚠

Warning: a staircase has no slope. "The tree predicts ₹150,000 at 6.5" contains no claim about the rate of salary growth — unlike Chapter 4's slope (₹9,346/year), the tree's local behaviour is a flat zero derivative everywhere except the walls, where it jumps. If your question is "how fast", ask an equation, not a tree.

4. Why Trees Never Need Scaling

Chapter 3's table promised a scale-free model class, and here it is — for a reason worth stating precisely, because it is an interview favourite. Every split is a comparison against a threshold: feature ≤ t. Rescale the feature — rupees to thousands of rupees, Celsius to Fahrenheit — and the tree simply learns the correspondingly rescaled threshold. The partition of the data (who goes left, who goes right) is unchanged, so the predictions are identical:

x ≤ t ⇔ (x/c) ≤ (t/c) — thresholds absorb units

Contrast the two families met so far: OLS (Ch 4–6) is scale-agnostic (exact algebra, scaling merely re-expresses coefficients), while kernels (Ch 7) are scale-hostile (distances mix units catastrophically). Trees are scale-blind: preprocessing can skip scaling entirely, and mixing a rupees column with a years column in one tree is perfectly safe.

That blindness is a genuine production convenience: no scalers to fit, no inverse transforms on predictions, no leakage risk from a mis-fitted scaler. One caveat for later: scale-blindness applies to how the tree splits, not to everything in its vicinity — when Chapter 9 averages trees with other tools, or when trees feed predictions downstream, Chapter 3's ritual still governs the pipeline.

5. Depth: The New Bias-Variance Dial

Uncontrolled, a regression tree grows until every leaf is pure — and with ten distinct levels that means ten leaves and R² = 1.0, the memorisation ceiling. Chapter 6's dial was polynomial degree; Chapter 7's was C, ε, γ; the tree's dial is max_depth — with close cousins min_samples_leaf (no leaf smaller than k rows) and min_samples_split. All real numbers, same data:

max_depth	Leaves	R² (in-sample)	Pred @6.5	Pred @12	Diagnosis
1	2	0.7764	₹124,375	₹750,000	One wall (root split ≤ 8.5): CEO vs everyone
2	4	0.9835	₹82,500	₹1,000,000	Too coarse near 6.5 — badly underfits the middle
3	6	0.9981	₹130,000	₹1,000,000	Steps aligning with the grid
None (source)	10	1.0000	₹150,000	₹1,000,000	Pure memorisation — every leaf one employee

Watch the @6.5 column swing ₹124k → ₹82k → ₹130k → ₹150k with depth — the negotiation's answer is a function of a hyperparameter, which is exactly why Chapter 31 will insist on tuning with held-out evidence rather than defaults. And reread the R² column against Chapter 6's lesson: in-sample R² climbing to a perfect 1.0 is not a triumph, it is the memorisation signature. On ten noiseless points the memoriser happens to be right; on noisy data the same tree would be confidently wrong.

💡

Pro Tip: The honest small-data instrument is the same as Chapter 6's: leave-one-out or cross-validated scoring per depth (Chapter 31's machinery). With clean monotone data the full tree survives; the professional habit is to check, not assume.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

The shortest pipeline in Part II — no scaler, no transformer, no reshaping. Load, fit, predict; the source adds a fine grid purely to draw the staircase:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:-1].values          # Level
y = dataset.iloc[:, -1].values            # Salary

from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(random_state=0)
regressor.fit(X, y)

print('R2    :', regressor.score(X, y))            # 1.0
print('@6.5  :', regressor.predict([[6.5]]))       # [150000.]
print('leaves:', regressor.get_n_leaves())          # 10

# Draw the staircase on a fine grid
X_grid = np.arange(min(X), max(X), 0.01).reshape(-1, 1)
plt.scatter(X, y, color='red')
plt.plot(X_grid, regressor.predict(X_grid), color='blue')
plt.title('Truth or Bluff (Decision Tree Regression)')
plt.xlabel('Position level'); plt.ylabel('Salary')
plt.show()

Listing 8.1 — The complete tree pipeline (source: decision_tree_regression.py).

6.1 Reading the fitted model itself

from sklearn.tree import plot_tree

plt.figure(figsize=(10, 6))
plot_tree(regressor, feature_names=['Level'], filled=True, rounded=True)
plt.show()

# Or walk the structure programmatically
t = regressor.tree_
print('root split: Level <= %.1f' % t.threshold[0])   # 8.5
print('depth      :', regressor.get_depth())           # 9

Listing 8.2 — Two ways to inspect the tree: draw it, or query the underlying arrays.

💡

Pro Tip: random_state=0 matters here even on deterministic data: when two splits tie on variance reduction, sklearn breaks the tie with feature order and a seeded search — fixing the seed keeps your tree reproducible run-to-run (Chapter 3's habit, now for grown models).

7. Reading the Tree — and Its Flat Horizon

7.1 The interpretability superpower

The real fitted tree's first three questions, translated to English:

"Is level ≤ 8.5?" — No (the ₹500k Senior Partner and ₹1M CEO branch off alone; this one split alone lifted R² to 0.78)
"Is level ≤ 4.5?" — separates the junior band (≤ ₹80k) from the middle band
"Is level ≤ 6.5?" — and there is our negotiation wall: 6.5 goes left, inheriting ₹150,000

That is the entire argument for trees in production: a compliance officer, a loan officer, a client can audit the logic. No other model in Part II can make that claim — SVR offers invisible bumps, polynomials offer exploding coefficients.

7.2 The flat horizon: trees cannot extrapolate

Ask the tree about level 12 — two steps past the CEO — and it answers ₹1,000,000: the CEO salary, forever. Every level beyond 10 falls into the rightmost leaf, and a leaf's prediction never changes. Complete the three-model comparison on extrapolation, all real numbers:

Model	@12 prediction	Behaviour beyond the data
Poly degree 4 (Ch 6)	₹3,035,985	Detonates — the x⁴ term owns the horizon
SVR default (Ch 7)	₹461,423	Decays toward a bounded value — cautious but invented
Tree (this chapter)	₹1,000,000	Perfectly flat — repeats the last leaf eternally

Three wrong answers in three different flavours — the common lesson stands: no model extrapolates honestly, but each fails characteristically, and knowing the flavour is part of knowing the model.

⚠

Warning — instability: trees are high-variance learners in Chapter 6's technical sense: resample the data, delete one row, and the greedily chosen splits can rearrange completely — a different tree with different walls and a different @6.5. A single tree's confidence is partly an artefact of its particular sample. This exact flaw is the launchpad for Chapter 9.

8. Strengths, Limits & the Road Ahead

Strengths	Limitations
Interpretable — the model is an auditable flowchart	Step-shaped predictions: no slopes, no rates
Scale-blind: no scaling, no transformers, no leakage risk	Unpruned trees memorise (R² = 1.0 on any distinct-valued feature)
Captures nonlinearity and feature interactions natively	High variance: small data changes grow different trees
Fast training, fast prediction, mixed feature types welcome	Cannot extrapolate — flat beyond the training range

The stage is set for Part II's finale. The tree's weaknesses — instability and memorisation — are both variance problems, and Chapter 2 long ago hinted at the cure for a noisy average: average many of them. A random forest grows a whole choir of de-correlated trees on random data slices and random feature subsets, and lets the ensemble vote with means. The instability remains in each tree; the forest cancels it in the average — bagging, the statistics of Chapter 9.

🛠 Hands-On: Grow, Prune, ReadBeginner

HR wants the negotiation settled with a model they can audit. Grow the tree, prune it to a defensible depth, extract the rules that decide level 6.5, and document what happens past level 10.

Steps

Fit the default tree; verify R² = 1.0 and count the leaves. Why exactly 10?
Refit with max_depth 1, 2, 3; tabulate R² and @6.5. Which depth would you defend to HR, and why not the deepest?
Print or draw the tree; write out the first three questions in English.
Predict at 6.5 (default tree) and explain the wall that decided it.
Predict at 12 and at 50. Explain the behaviour in one sentence.
Compare the tree's answers with Chapter 6's polynomial and Chapter 7's SVR in a three-row table — fit, @6.5, @12.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor, plot_tree
import matplotlib.pyplot as plt

ds = pd.read_csv('Position_Salaries.csv')
X = ds.iloc[:, 1:-1].values.astype(float)
y = ds.iloc[:, -1].values.astype(float)

for d in [1, 2, 3, None]:
    r = DecisionTreeRegressor(random_state=0, max_depth=d).fit(X, y)
    print('depth %-4s R2=%.4f  @6.5=%7.0f  @12=%7.0f  @50=%7.0f  leaves=%d' %
          (str(d), r.score(X, y), r.predict([[6.5]])[0],
           r.predict([[12.0]])[0], r.predict([[50.0]])[0], r.get_n_leaves()))

full = DecisionTreeRegressor(random_state=0).fit(X, y)
print('root split: Level <=', full.tree_.threshold[0])     # 8.5
plot_tree(full, feature_names=['Level'], filled=True)
plt.show()

Expected results (real):

Step 1: R² = 1.0000, 10 leaves — ten distinct levels, unlimited depth isolates each one; the leaf predictions are exactly the ten observed salaries.
Step 2: depth 1 → R² 0.7764, @6.5 ₹124,375; depth 2 → 0.9835, ₹82,500 (badly wrong near 6.5!); depth 3 → 0.9981, ₹130,000; full → 1.0, ₹150,000. Defence: on this clean grid depth 3 or full both work in-sample — the honest answer names the missing evidence (cross-validation, Ch 31) rather than trusting R² = 1.0.
Step 3: ≤ 8.5? → ≤ 4.5? → ≤ 6.5? (Section 7's list).
Step 4: ₹150,000 — 6.5 satisfies ≤ 6.5, so it falls in the level-6 leaf.
Step 5: @12 = @50 = ₹1,000,000 — flat forever; the rightmost leaf never updates.
Step 6: fit: 0.997 / 0.752 / 1.0 (poly / SVR / tree); @6.5: 158,862 / 170,370 / 150,000; @12: 3.04M / 461k / 1.0M — three philosophies, one ballpark for 6.5, three characteristic failures at 12.

🎯 Key Takeaways

A regression tree grows by greedy axis-aligned splits that maximise variance reduction — isolate the outliers first, always.

Leaves predict the mean of their residents; the fitted "curve" is a staircase — flat inside each box, jumping at each wall.

Thresholds absorb units, so trees are scale-blind: no scalers, no inverse transforms, mixed feature types welcome.

max_depth is the bias-variance dial: real run moved @6.5 across ₹82k–₹150k; unlimited depth reached R² = 1.0 — the memorisation signature.

Trees cannot extrapolate: beyond the range they repeat the last leaf eternally — ₹1,000,000 at every level past 10.

Interpretability is the superpower; instability is the price — one resample regrows a different tree. Chapter 9's forests average the instability away.