Part X — Model Selection & Boosting

Chapter 32 · XGBoost & Gradient Boosting

Chapter 17 grew a forest of independent trees and averaged them; this chapter grows trees that argue — each new one trained on the mistakes of all before it. Gradient boosting is the idea, and XGBoost is its decade-defining engineering: for years, the winner of most tabular competitions on Kaggle was some flavour of it. The method is built here by hand on four data points — base prediction, residuals, a one-split stump, a learning rate, three rounds, every number exact — before the real run on 683 breast-cancer biopsies, where parameter plateaus, early stopping at round 107 of 1000, and an identifier column that "improves" one split but not the cross-validation all become lessons.

📖 Chapter 32 of 33 ⏱ ~90 min read Mostly Core Stable Volatile 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Place boosting beside bagging: sequential corrective trees fitting residuals, versus parallel independent trees averaged
Run gradient boosting by hand on four points: F₀ = mean, residuals, a stump, η = 0.5, three rounds — MSE 2.5 → 0.8125 → 0.3906, every number exact
Explain the mathematics: residuals are negative gradients, boosting is gradient descent in function space, and small η plus many trees walks carefully
State what makes XGBoost XGBoost: second-order steps, regularised objective, sparsity-aware splits, serious engineering
Run the real breast-cancer pipeline: modern label handling, the ID-column honesty test, the scoreboard against random forest and logistic regression
Read parameter behaviour from real sweeps — the n_estimators plateau, the learning-rate cliff, early stopping at 107 of 1000 — and the feature importances

1. The Third Ensemble Idea

Chapter 17 taught the forest's trick: grow many independent trees on random samples of rows and features, then average — errors cancel, variance falls. Boosting is the other half of the ensemble world, and its opposite in every design choice:

	Bagging (Ch 17, random forest)	Boosting (this chapter)
Tree order	parallel — no tree sees another	sequential — each tree sees the previous trees' errors
Tree target	the labels, on a bootstrap sample	the residuals — what the ensemble still gets wrong
Tree size	deep, low-bias, high-variance individuals	shallow "weak learners" (stumps by default)
Combination	average / majority vote	running sum, each tree scaled by learning rate η
Attacks	variance	bias first, then variance — gradually, additively
Failure mode	plateaus (more trees don't hurt)	can overfit — more trees can hurt without early stopping

The running dataset: Data.csv — the classic Wisconsin breast-cancer screen. 683 biopsies, nine morphology features scored 1–10 by a pathologist (clump thickness, uniformity of cell size and shape, marginal adhesion, epithelial cell size, bare nuclei, bland chromatin, normal nucleoli, mitoses), and a Class label: 2 = benign (444) · 4 = malignant (239). The stakes make Chapter 18's metric discussion concrete again: a missed malignancy is not the same error as a false alarm.

📝

Note: one modernisation is mandatory before any code runs: current XGBoost rejects labels {2, 4} outright ("Invalid classes… Expected: [0 1], got [2 4]" — the real error). The listing maps 2 → 0 (benign) and 4 → 1 (malignant) first; the source predates this requirement.

2. Gradient Boosting, by Hand

Four points, regression (boosting's clearest classroom), one stump per round, learning rate η = 0.5 — every number below exact and verified:

data: (1, 1) · (2, 2) · (3, 4) · (4, 5)

A near-line with a bend — no single stump can fit it, but a sum of stumps will.

2.1 Round 0 — the base model

The starting ensemble is the best constant: the mean. F₀ = (1 + 2 + 4 + 5)/4 = 3 for every point. Its residuals (truth minus current prediction): 1 − 3 = −2, 2 − 3 = −1, 4 − 3 = +1, 5 − 3 = +2, with mean squared error (4 + 1 + 1 + 4)/4 = 2.5.

2.2 Round 1 — a stump fits the residuals

The first weak learner never looks at y. It looks at the residuals [−2, −1, +1, +2] and finds the best one-split fit: split between x = 2 and x = 3, leaf values = leaf means — left −1.5, right +1.5. The ensemble updates by a fraction η = 0.5 of the correction:

F₁(x) = 3 + 0.5 × (−1.5) = 2.25 for x ≤ 2 · 3 + 0.5 × (+1.5) = 3.75 for x > 2

New residuals: 1 − 2.25 = −1.25, 2 − 2.25 = −0.25, 4 − 3.75 = +0.25, 5 − 3.75 = +1.25 — and MSE drops to (1.5625 + 0.0625 + 0.0625 + 1.5625)/4 = 0.8125.

2.3 Rounds 2 and 3 — the correction shrinks

Each new stump fits the new residuals, whose leaf means are now (−1.25 − 0.25)/2 = −0.75 and +0.75. Same split — the pattern of error outlives each correction, only smaller:

Round	Predictions F(x)	Residuals	Stump leaves	MSE
0	3, 3, 3, 3	−2, −1, +1, +2	—	2.5000
1	2.25, 2.25, 3.75, 3.75	−1.25, −0.25, +0.25, +1.25	−1.5 / +1.5	0.8125
2	1.875, 1.875, 4.125, 4.125	−0.875, +0.125, −0.125, +0.875	−0.75 / +0.75	0.3906
3	1.6875, 1.6875, 4.3125, 4.3125	−0.6875, +0.3125, −0.3125, +0.6875	−0.375 / +0.375	0.2852

Figure 32.1 — The hand example: constant base (dashed), the round-1 step (solid), the round-3 step (dashed green) closing onto the four points. MSE 2.5 → 0.285 in three rounds of shrinking corrections.

Read the pattern: each round's corrections are (roughly) half the previous round's, and the MSE roughly halves with them — 2.5 → 0.8125 → 0.3906 → 0.2852. The ensemble is not learning four independent models; it is walking, in η-sized steps, toward the data. And notice what η = 0.5 bought: with η = 1 the first round would already jump to 1.5 and 4.5 — faster here, but on noisy data full-size steps overshoot and memorise. Section 3 makes that intuition precise.

💡

Pro Tip — the whole algorithm in one sentence: start with the mean; repeatedly fit a small tree to what the current ensemble still gets wrong; add a fraction η of that tree to the ensemble. Everything else in this chapter is that sentence, industrialised.

3. Gradient Descent in Function Space

Why "residuals"? For squared loss L = ½(y − F(x))², the derivative with respect to the prediction F is −(y − F(x)) — so the residual is exactly the negative gradient of the loss. Fitting each new tree to residuals is taking a gradient step — not in the space of parameters (Chapter 26's δ), but in the space of functions: the tree h(x) is the step direction.

Fᵢ₊₁(x) = Fᵢ(x) + η · hᵢ₊₁(x)

h = a small tree fitted to the current negative gradients (residuals for squared loss). For classification — this chapter's real task — the same machinery runs on the log-odds: F₀ = log(p/(1−p)) of the class balance, each tree steps the log-odds, and the sigmoid (Chapter 11) turns the sum back into a probability. Loss: log-loss; its negative gradient is the familiar y − p, the residual in probability clothing.

Two consequences of the gradient view organise everything that follows. The learning rate is a step size: small η with many trees is gradient descent with a cautious step — slower, smoother, less likely to leap onto noise (Chapter 26 said the same about SGD). The loss is swappable: squared loss, log-loss, ranking losses — the algorithm only ever needs the loss's gradient and a tree to fit it. That generality is why "gradient" boosting displaced every earlier boosting recipe.

📝

Note — boosting's bias-variance story: bagging attacked variance by averaging independent, high-variance trees. Boosting starts with a high-bias base (a constant!) and removes bias round by round; variance is what creeps back if the rounds run too long. Hence the chapter's later tension: n_estimators up helps until it doesn't, and early stopping is the referee.

4. What Makes XGBoost XGBoost

Gradient boosting was folklore by the 2000s; XGBoost ("eXtreme Gradient Boosting", 2014) made it fast, regularised, and reliable enough to dominate tabular machine learning for a decade. Four upgrades matter conceptually:

Second-order steps. Plain gradient boosting uses the loss's slope (first derivative). XGBoost also uses its curvature (second derivative) — Newton-style steps that know not just which way but how far, converging in fewer, better-placed trees.
A regularised objective. Each split must earn its keep against an explicit complexity penalty Ω = γ·(leaf count) + ½λ·Σ(weight²): γ prunes leaves that don't reduce loss enough, λ shrinks leaf values. Chapter 6's regularization lesson, built into the tree's growth rule itself.
Sparsity awareness. Missing values get a learned default direction per split — the model trains through them natively, no imputation required (Chapter 3's toolbox, one entry lighter).
Engineering. Histogram-based split finding, column block storage, parallel and cache-aware execution — the "extreme" in the name is mostly systems engineering, and it is why a thousand trees on a million rows is a coffee-break job, not a weekend one.

🌎

Real World: from roughly 2015 on, "what won the tabular competition?" had a boring answer: gradient boosting — XGBoost first, then LightGBM and CatBoost. Credit scoring, demand forecasting, click-through prediction, bioinformatics: the same engineering, winning everywhere tables live. The 683-row biopsy set below is exactly the shape of problem it owns.

📋 Stable content — Reviewed: October 2026

5. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, 1:-1].values        # 9 morphology features
                                        # (drop Sample code number — an ID)
y = (dataset.iloc[:, -1].values == 4).astype(int)
                                        # Class 2/4 -> 0/1: modern XGBoost
                                        # rejects labels {2, 4} outright

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

from xgboost import XGBClassifier       # pip install xgboost
classifier = XGBClassifier(eval_metric='logloss')
classifier.fit(X_train, y_train)

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
print(confusion_matrix(y_test, y_pred))    # [[85  2]
                                           #  [ 2 48]]
print(accuracy_score(y_test, y_pred))      # 0.971

# Chapter 31's discipline, applied immediately
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(classifier, X_train, y_train, cv=10)
print("Accuracy: {:.2f} %".format(accuracies.mean() * 100))   # 96.71 %
print("Std:      {:.2f} %".format(accuracies.std()  * 100))   # 1.96 %

Listing 32.1 — The complete pipeline (source: xg_boost.py, modernised: label mapping, ID column dropped, eval_metric set).

# The idioms that matter

# early stopping: let a validation set decide when to stop growing
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train,
                                            test_size=0.2, random_state=0)
model = XGBClassifier(n_estimators=1000, learning_rate=0.1,
                      early_stopping_rounds=20, eval_metric='logloss')
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
print(model.best_iteration)        # 107 — 1000 requested, 107 needed

# the dials (Section 6 sweeps them for real)
XGBClassifier(n_estimators=100, learning_rate=0.3, max_depth=6,
              subsample=1.0, colsample_bytree=1.0)   # library defaults

# what the model learned
classifier.feature_importances_    # gain-based, per feature

Listing 32.2 — Early stopping, the parameter cupboard, and importances.

⚠

Warning — trees are not scale-free the other way: XGBoost needs no feature scaling (trees split on thresholds), so the StandardScaler of Parts III and IX disappears here. But it still needs the label encoding fix, and it will happily memorise an identifier column if you leave one in — Section 6 shows that experiment, run for real.

6. The Real Run — 683 Biopsies

Split 546 train / 137 test (87 benign, 50 malignant), seed 0, library-default hyperparameters. The source-faithful run (which leaves the ID column in — Chapter 3's rule bent) versus the clean run, both real:

Variant	Confusion matrix	Test acc	10-fold CV
Source-faithful (ID column left in)	[[85, 2], [1, 49]]	0.978	0.9671 ± 0.0255
Clean (ID dropped)	[[85, 2], [2, 48]]	0.971	0.9671 ± 0.0196

A teaching gift, courtesy of the data: the identifier column "improves" the one-split test score (0.978 vs 0.971 — one fewer miss) despite carrying zero information — a patient ID cannot predict a tumour. Chapter 31's lesson, demonstrated rather than asserted: the cross-validation means are identical (0.9671 — the ID adds nothing on average), and the clean model's fold-spread is tighter (± 0.0196 vs ± 0.0255). Single-split wins on noise features are luck; CV prices them at zero.

6.1 The scoreboard

Model (same clean split)	Confusion matrix	Test acc	10-fold CV
XGBoost (defaults)	[[85, 2], [2, 48]]	0.971	0.9671 ± 0.0196
Random forest (Ch 17)	[[84, 3], [1, 49]]	0.971	0.9670 ± 0.0258
Logistic regression (Ch 11)	[[84, 3], [3, 47]]	0.956	—

The honest read: XGBoost and the forest tie on the test set and in CV — with 137 test rows, they differ by one or two patients. What separates them is character, not score: the forest misses one malignancy and raises three false alarms; XGBoost, two and two (precision and recall both 0.960). Logistic regression — nine features, mostly monotone relationships with the label — was never going to be far behind at 0.956. On a dataset this tidy, Chapter 31's verdict repeats: defaults sit on the plateau; structure barely matters.

6.2 The dials, swept for real

n_estimators (lr default)	10	50	100	200	400
Test acc	0.9635	0.9708	0.9708	0.9708	0.9708

learning_rate (n = 100)	0.05	0.1	0.3	0.9
Test acc	0.9708	0.9708	0.9708	0.9635

max_depth	2	3	6	10
Test acc	0.9635	0.9635	0.9708	0.9708

Chapter 31's plateau, seen again from a new angle: from 50 trees on, nothing moves — 0.9708 from n = 50 through n = 400. The learning rate is flat until 0.9, where the over-large step finally costs a patient (0.9635 — Section 3's overshoot, live). Depth 2–3 stumps are slightly weak here (0.9635); the default 6 suffices; 10 adds nothing. On messier data these curves would climb and fall; on this tidy set they run flat — which is itself the diagnosis: the model class had already learned the problem by round 50.

And the referee, run for real: a 1000-tree model with early_stopping_rounds=20 on a validation split stops at best_iteration = 107 — validation log-loss 0.0535, test accuracy 0.978 (the early-stopped model actually edges the fixed-100 default by one patient). Request 1000, need 107: early stopping is not an optimisation nicety, it is how boosting knows when to stop walking.

6.3 What the trees learned

Feature (gain importance)	Share
Uniformity of Cell Size	0.437
Uniformity of Cell Shape	0.211
Single Epithelial Cell Size	0.151
Bare Nuclei	0.087
Bland Chromatin	0.046
Clump Thickness · Normal Nucleoli · Marginal Adhesion	0.028 · 0.022 · 0.019
Mitoses	0.000

The model's attention matches a century of pathology: cell size and shape uniformity carry two-thirds of the diagnostic weight (0.437 + 0.211) — cancer is cells growing inconsistently — while Mitoses gets exactly zero: by the time the other eight features have spoken, mitotic activity adds nothing the ensemble needs. Chapter 17's importances were a near-tie between two features; boosting's gains concentrate far harder on the few features that pay. (Whether that concentration is trust or fragility is exactly what SHAP-style attribution — Section 8 — was invented to interrogate.)

7. XGBoost in the Ecosystem

The course's last algorithm lands in a crowded toolbox. The choosing rules, earned over thirty-two chapters:

Problem shape	Reach for	Why (course evidence)
Tabular, any size, mixed features	Gradient boosting first	Chapter 31–32: defaults on plateaus, CV-tight, handles missing values, no scaling
Tabular, need interpretability	Logistic / single tree / LDA	Chapters 11, 16, 29: coefficients, rules, two readable axes
Images, audio, text	CNNs / transformers	Chapters 25–27: no spatial/locality structure for trees to grab
Very little data	Simple models	Chapters 26, 31: boosting's rounds memorise; bias-variance bites
Quick baseline, no tuning budget	Random forest	Chapter 17: nearly un-tunable, no early stopping needed

One more practice note worth the ink: boosting's sequential nature makes its errors correlated — late trees correct early trees, so unlike a forest's independent voters, a boosted ensemble can be confidently wrong together on genuinely novel inputs. The standard mitigations are already in the API: subsample (each tree sees a random fraction of rows) and colsample_bytree (a random fraction of features) re-inject the diversity that sequential fitting strips out — "stochastic gradient boosting", Chapter 17's row/column sampling reborn inside the other ensemble.

💡

Pro Tip — the starting recipe: n_estimators = 1000 with early_stopping_rounds = 50, learning_rate = 0.05–0.1, max_depth = 6 — then let Chapter 31's grid search argue about the rest. On this dataset that recipe stops at 107 trees and matches everything else on the scoreboard.

⚠ Volatile content — Reviewed: October 2026 · Next review: January 2027

8. The Road Ahead

Where the boosting world is moving — the volatile neighbourhood, reviewed quarterly:

The sibling engines: LightGBM (histogram-first, leaf-wise growth — often the fastest on large tables) and CatBoost (ordered boosting, native categorical handling). Same gradient mathematics; different engineering bets. The XGBoost skills transfer one-to-one.
Boosting vs deep on tabular: a live debate. Tree ensembles remain the default champion for moderate-sized tables; deep tabular architectures keep attacking the large-data and multimodal ends. Chapter 26's honest ledger still holds: match the tool to the data shape.
Interpretability: SHAP values (game-theoretic per-prediction attribution, built on the tree structure) turn "the model looked at cell size" into "this patient's score came 60% from cell-size uniformity" — the audit trail regulators increasingly demand.
Next and last: Chapter 33 — the capstone. One end-to-end pipeline — raw data, preprocessing, shortlist, honest comparison, business metric — welding all thirty-two chapters into the professional workflow this course has been building toward.

🛠 Hands-On: Boost, Diagnose, DefendIntermediate

The oncology group wants a malignancy screener and one tough question answered: is XGBoost worth its complexity over the random forest already in place? Build, verify the hand arithmetic, sweep, and defend the recommendation in five sentences.

Steps

Hand-verify Section 2 completely: F₀ = 3, round-1 residuals and stump, η = 0.5 updates, and the MSE sequence 2.5 → 0.8125 → 0.3906 → 0.2852. Then reproduce it in code.
Run the label mapping experiment: fit with classes {2, 4} and read the real error; map to {0, 1} and proceed.
Run source-faithful (ID column in) vs clean: reproduce both confusion matrices and both CV summaries; explain the ID's phantom test "gain".
Reproduce the scoreboard (XGBoost vs random forest vs logistic) with precision/recall alongside accuracy.
Sweep n_estimators, learning_rate, max_depth; reproduce the plateaus and the lr = 0.9 dip.
Run early stopping (n = 1000, lr = 0.1, patience 20); verify best_iteration ≈ 107 and its test accuracy.
Bonus: refit with subsample = 0.8 and colsample_bytree = 0.8; report whether stochastic boosting moves anything on this dataset.

View Solution / Walkthrough

import numpy as np, pandas as pd, xgboost as xgb
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import (confusion_matrix, accuracy_score,
                             precision_score, recall_score)

# ---- step 1: the hand example, reproduced
ys = np.array([1., 2., 4., 5.]); F = np.full(4, ys.mean())
for r in range(3):
    res = ys - F
    print('round', r, 'MSE %.4f' % (res**2).mean(),
          'leaves %.3f / %.3f' % (res[:2].mean(), res[2:].mean()))
    F = F + 0.5 * np.where(np.arange(4) < 2, res[:2].mean(), res[2:].mean())
print('final MSE %.4f' % ((ys - F)**2).mean())       # 0.2852

# ---- steps 2-4
ds = pd.read_csv('Data.csv')
y = (ds.iloc[:, -1].values == 4).astype(int)
for cols, tag in [(slice(0, -1), 'ID incl'), (slice(1, -1), 'clean')]:
    X = ds.iloc[:, cols].values
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.2,
                                          random_state=0)
    m = xgb.XGBClassifier(eval_metric='logloss').fit(Xtr, ytr)
    yp = m.predict(Xte)
    cv = cross_val_score(xgb.XGBClassifier(eval_metric='logloss'),
                         Xtr, ytr, cv=10)
    print(tag, confusion_matrix(yte, yp).tolist(),
          'acc %.4f | CV %.4f ± %.4f' % (
          accuracy_score(yte, yp), cv.mean(), cv.std()))

# ---- steps 5-6
X = ds.iloc[:, 1:-1].values
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.2, random_state=0)
for n in [10, 50, 100, 200, 400]:
    print('n=%d acc %.4f' % (n, accuracy_score(yte,
          xgb.XGBClassifier(n_estimators=n, eval_metric='logloss')
          .fit(Xtr, ytr).predict(Xte))))
Xt, Xv, yt, yv = train_test_split(Xtr, ytr, test_size=.2, random_state=0)
m = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.1,
                      early_stopping_rounds=20,
                      eval_metric='logloss').fit(Xt, yt,
                      eval_set=[(Xv, yv)], verbose=False)
print('best_iteration', m.best_iteration,               # 107
      'test acc %.4f' % m.score(Xte, yte))              # 0.978

# ---- step 7
ms = xgb.XGBClassifier(subsample=0.8, colsample_bytree=0.8,
                       eval_metric='logloss').fit(Xtr, ytr)
print('stochastic acc %.4f' % accuracy_score(yte, ms.predict(Xte)))

Expected results (real): hand MSE 2.5 → 0.8125 → 0.3906 → final 0.2852 exactly; labels {2, 4} rejected with "Invalid classes… Expected: [0 1]"; ID-included cm [[85, 2], [1, 49]] 0.978, CV 0.9671 ± 0.0255; clean cm [[85, 2], [2, 48]] 0.971, CV 0.9671 ± 0.0196; forest 0.971 (CV 0.9670 ± 0.0258), logistic 0.956; sweeps flat at 0.9708 from n = 50, lr = 0.9 dips to 0.9635, depth 2–3 at 0.9635; early stop at iteration 107 with test 0.978; importances Cell Size 0.437 / Shape 0.211 / Mitoses 0.000. Recommendation sketch: keep the forest or adopt XGBoost — scores tie; adopt XGBoost if missing values, larger data, or staged tuning are on the roadmap, since its early stopping and regularised growth scale gracefully.

🎯 Key Takeaways

Boosting is bagging's opposite: sequential shallow trees fitted to previous trees' residuals, summed with learning rate η — correction, not averaging.

Hand-verified: F₀ = 3, stump at ∓1.5, η = 0.5 — MSE 2.5 → 0.8125 → 0.3906 → 0.2852. Boosting walks to the data in shrinking steps.

Residuals are negative gradients: boosting is gradient descent in function space. Small η + many trees = careful steps; XGBoost adds curvature (Newton steps) and regularised splits.

Real run: 0.971 on 683 biopsies (CV 0.9671 ± 0.0196), tying the random forest — and the ID column's phantom 0.978 one-split "win" is exposed by identical CV means.

The dials plateau: 0.9708 from 50 to 400 trees; lr 0.9 finally overshoots (0.9635); early stopping halts a 1000-tree request at 107 — and still scores 0.978.

The trees concentrate: cell-size uniformity 0.437, shape 0.211, Mitoses exactly 0.000 — boosting's gains focus where the signal is. Default for tabular data; not for images or text.