Part X — Model Selection & Boosting

Chapter 31 · Model Selection: k-Fold Cross-Validation & Grid Search

Thirty chapters built models; several turned dials by hand — a degree here, a C there, a gamma in two chapters running. This chapter makes the choosing honest. First the uncomfortable evidence: the very same SVM, evaluated on ten different random splits of the same data, returns accuracies from 88% to 95% — Chapter 18's neat one-split scoreboard was a lottery ticket. k-fold cross-validation replaces the lottery with an average over every row's turn as a test case — computed here fold by fold on real data, pooled by hand into 271 of 300. Then grid search automates the dial-turning: 40 parameter candidates × 10 folds = 400 real fits, whose six-way tie for first delivers the chapter's most valuable and least advertised lesson.

📖 Chapter 31 of 33 ⏱ ~85 min read Mostly Core Stable Volatile 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Demonstrate, on real data, that a single train/test split can swing accuracy by 7 points — and why every earlier scoreboard needs this asterisk
Run k-fold cross-validation by hand: ten real folds, every row held out exactly once, pooled 271/300 = 90.33%
Read the two-number summary honestly — mean 90.33% as the estimate, std 6.57% as the uncertainty — and compare 5-fold, 10-fold, LOOCV
Explain stratified folds, and why the scaler must live inside the pipeline, not before it
Execute a real 40-candidate × 10-fold grid search: interpret the six-way tie, the linear-vs-rbf gap, and the tuned model that gained nothing on the test set
State the rules of honest tuning: test once, CV inside train, pick the business metric, beware overfitting the validation set

1. The Luck of One Split

The running setup is old friends by now: Social_Network_Ads.csv — 400 customers, Age and Estimated Salary, 143 buyers — split 75/25, standardised, and Chapter 14's kernel SVM (RBF) as the model. On the seed-0 split it scores 93% (cm [[64, 4], [3, 29]] — Chapter 18's scoreboard line). A fair question with an uncomfortable answer: how much of that 93% is the model, and how much is the split?

The experiment: the identical model and pipeline, retrained and retested on ten different random 75/25 splits (seeds 0–9). The real accuracies:

Figure 31.1 — Same model, ten random splits (seeds 0–9): 0.93, 0.88, 0.90, 0.89, 0.92, 0.93, 0.93, 0.91, 0.91, 0.95. The model never changed; the number on the report did.

0.880 to 0.950 — a seven-point spread on the strength of a random seed. With 100 test customers, one person is one percentage point; the lottery band is seven customers. Chapter 18's careful eight-model comparison was built on one such draw (and its conclusions survived — the gaps there were larger than the noise — but that was fortune, not design). The professional answer to single-split luck is to stop asking one question and start asking every question the data can answer about itself.

📝

Note: Chapter 10 already met k-fold cross-validation for regression (the linear model's honest 0.922 ± 0.050 on 50_Startups). This chapter rebuilds the method from the fold level up on the classification side, then pairs it with its constant companion: grid search.

2. k-Fold Cross-Validation, by Hand

The idea in one sentence: cut the training set into k pieces ("folds"), hold out each piece in turn, train on the remaining k − 1, test on the held-out piece, and average the k scores. Every row takes exactly one turn as a test case, and no row is scored by a model that trained on it.

With k = 10 on the 300 training rows: ten folds of 30, and since cross_val_score on a classifier defaults to stratified folds, each fold carries (almost) the same buyer ratio as the whole training set — 111 buyers across 300 rows → nine folds of 11 buyers + 19 non-buyers, one fold of 12 + 18. Stratification keeps a lucky fold from being merely an easy-class fold. The ten real fold accuracies:

Fold	1	2	3	4	5	6	7	8	9	10
Buyers / non	11/19	11/19	11/19	11/19	11/19	11/19	11/19	11/19	11/19	12/18
Correct / 30	24	29	24	29	26	26	27	28	30	28
Accuracy	0.800	0.967	0.800	0.967	0.867	0.867	0.900	0.933	1.000	0.933

The hand arithmetic, in the order a professional would do it. First the pooled check — because every row is tested exactly once, the fold accuracies are just one big out-of-sample count in disguise:

24 + 29 + 24 + 29 + 26 + 26 + 27 + 28 + 30 + 28 = 271 correct of 300 ⇒ 271 / 300 = 0.9033

Which equals the mean of the ten fold accuracies — with equal fold sizes, averaging folds IS pooling rows. The CV score is honest at the row level: 271 customers were correctly classified by models that had never seen them.

Then the spread — the standard deviation, by hand (population form, dividing by k = 10, which is what accuracies.std() reports):

deviations from 0.9033: −0.103, +0.063, −0.103, +0.063, −0.037, −0.037, −0.003, +0.030, +0.097, +0.030
sum of squares = 0.0432 ⇒ 0.0432 / 10 = 0.00432 ⇒ √0.00432 = 0.0657

The two-number summary — 90.33% ± 6.57% — is the source's printout, reproduced by hand. Six and a half points of fold-to-fold wobble, from data that never changed: the folds themselves are samples, and samples scatter (Section 1's lesson, one level down).

Figure 31.2 — The ten real folds: 0.800 to 1.000. Fold 9 (green) was a perfect test set; folds 1 and 3 were the hardest customers. The mean and its ±1-std band tell the honest story.

⚠

Warning — the scaler's place: this CV was run on pre-scaled training data, as the source does. The disciplined version wraps scaler + model in a Pipeline so each fold's scaler fits on that fold's 9 training parts only (Listing 31.2). On this two-feature dataset both routes return 90.33% ± 6.57% — but with leakage-prone preprocessing (Chapters 25, 28), the pipeline route is the only honest one.

3. Reading the Numbers: Mean and Spread

The two numbers answer different questions. The mean (90.33%) estimates the model's true performance — an out-of-sample score computed from 271 independent verdicts, far sturdier than any single split of Figure 31.1. The std (6.57%) measures how much that estimate depends on which rows landed in which fold — the wobble of the measurement itself. Report them together, always: "90.33% ± 6.57%" is a finding; "90.33%" is a claim.

The fold count k is itself a choice, with a real trade-off:

Scheme	Fits	Train rows per fit	Mean	Std	Notes
5-fold	5	240	0.9033	0.0386	cheaper; less training data per fit (mild pessimism); steadier-looking std
10-fold	10	270	0.9033	0.0657	the professional default; more of each row's information used per fit
LOOCV (k = n)	300	299	0.9033	—	leave-one-out: 300 fits, mean 0.9033 here — maximum data per fit, but each fit's test is one row, so per-fold scores are nearly coin-flips and the compute bill is n model fits

The means agree at 90.33% across all three schemes on this data — reassuring, and typical for a stable model on a tidy dataset. In general: smaller k trains each fit on less data (slightly pessimistic bias); larger k makes the folds — and therefore the fold scores — more similar (their average can get noisier per unit of compute). Five and ten are the standard compromise, and 10 is the course's choice.

💡

Pro Tip — which k for which n: small datasets (hundreds of rows) favour 10-fold or LOOCV, because every row of training data matters; large datasets (tens of thousands+) often use 5-fold or even 3-fold, because each fold is already statistically ample and compute is the binding constraint.

4. Grid Search — 400 Real Fits

Cross-validation measures one fixed model; grid search uses it to choose. List the hyperparameter combinations worth trying, give each the full k-fold evaluation, and keep the winner. The source's grid, verbatim in structure:

Branch	Parameters	Values	Candidates
linear kernel	C	0.25, 0.5, 0.75, 1	4
RBF kernel	C × γ	4 C-values × 9 γ-values (0.1…0.9)	36
Total	evaluated by 10-fold CV each		40 × 10 = 400 fits

The real results:

Candidate (kernel, C, γ)	CV accuracy
rbf, C = 0.5, γ = 0.6 — reported winner	0.9067
rbf, C = 0.5, γ = 0.7 · rbf, C = 0.5, γ = 0.9 · rbf, C = 0.75, γ = 0.8 · rbf, C = 0.75, γ = 0.9 · rbf, C = 1, γ = 0.7	0.9067 — all tied
default (rbf, C = 1, γ = "scale") for comparison	0.9033
best linear: C = 0.25	0.8233
worst overall: linear, C = 1	0.8133

Three findings, each worth a paragraph.

The winner is a six-way tie. Six different (C, γ) combinations scored exactly 0.9067 — 272 of 300 rows. The optimum is not a needle; it is a plateau. Grid search must report one "best", and it reports the first it found — but treating that combination as meaningfully better than its five co-champions (or than most of the RBF neighbourhood) is reading noise as signal. When the top of the grid is flat, any resident of the plateau is fine; prefer the simplest.

The kernel choice dwarfs the dial-tuning. Every RBF candidate clears 0.89; every linear candidate sits near 0.82. The 9-point linear-vs-RBF gap is the structural decision (Chapter 14's kernel menu, re-derived honestly), while C and γ shuffle customers within the plateau. Lesson for grid design: spend candidates on structure (kernel, model family) before fine-tuning.

The tuned model gained nothing on the test set. Refit on all 300 training rows with the winning parameters, the model scores [[64, 4], [3, 29]] — 0.930 — identical to the default. The CV gain was 0.9033 → 0.9067 (one customer); the test gain was zero. This is the most common honest outcome of grid search on well-chosen defaults: confirmation, not transformation. The search's value was proving the default sat on the plateau — an answer you cannot get without running it.

🌎

Real World: production tuning loops look exactly like this — CV on training data, pick, refit, test once — but the grids are larger, the scoring matches the business metric, and the compute budget is negotiated. What rarely changes is the shape of the result: a few structural hyperparameters matter enormously, the rest decorate a plateau.

📋 Stable content — Reviewed: October 2026

5. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.svm import SVC
classifier = SVC(kernel='rbf', random_state=0)
classifier.fit(X_train, y_train)

# ---- k-fold cross-validation on the TRAINING set
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator=classifier, X=X_train, y=y_train, cv=10)
print("Accuracy: {:.2f} %".format(accuracies.mean() * 100))   # 90.33 %
print("Std:      {:.2f} %".format(accuracies.std() * 100))    # 6.57 %

# ---- grid search over kernel / C / gamma
from sklearn.model_selection import GridSearchCV
parameters = [{'C': [0.25, 0.5, 0.75, 1], 'kernel': ['linear']},
              {'C': [0.25, 0.5, 0.75, 1], 'kernel': ['rbf'],
               'gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]}]
grid_search = GridSearchCV(estimator=classifier,
                           param_grid=parameters,
                           scoring='accuracy',
                           cv=10,
                           n_jobs=-1)          # all CPU cores
grid_search.fit(X_train, y_train)
print("Best Accuracy: {:.2f} %".format(grid_search.best_score_ * 100))  # 90.67 %
print("Best Parameters:", grid_search.best_params_)
# {'C': 0.5, 'gamma': 0.6, 'kernel': 'rbf'}

# the tuned model, refit on all training data, evaluated once on test
best_accuracy = grid_search.score(X_test, y_test)                       # 0.93

Listing 31.1 — The complete model-selection pipeline (source: grid_search.py + k_fold_cross_validation.py).

# The idioms that make it honest

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), SVC(kernel='rbf', random_state=0))
accuracies = cross_val_score(pipe, X_train_RAW, y_train, cv=10)
# scaler fits inside each fold — leakage impossible by construction

pd.DataFrame(grid_search.cv_results_)[
    ['params', 'mean_test_score', 'std_test_score', 'rank_test_score']
].sort_values('rank_test_score')       # every candidate's full audit trail

grid_search.best_estimator_            # already refit on ALL training rows
GridSearchCV(..., scoring='f1')        # match the metric to the business question
RandomizedSearchCV(..., n_iter=40)     # sample the grid when it is vast

Listing 31.2 — Pipeline CV, the results audit trail, and the scoring/n_iter knobs.

💡

Pro Tip: refit=True (the default) means the returned object already retrained on the full training set with the winning parameters — best_estimator_ is ready to deploy, no second fit needed. And n_jobs=-1 is free speed: the 400 fits are independent, so they spread across every core you own.

6. The Rules of Honest Tuning

Everything in this chapter compresses into six rules. Break any one and the number on your report becomes fiction:

The test set is touched once. All CV, all grid search, all comparison happens inside the training set. The test set appears at the very end, for the final number. (Tuning on the test set is the cardinal sin — it turns the umpire into a coach.)
Preprocessing lives inside the fold. Wrap scaler + model in a Pipeline so each fold fits its own scaler on its own training part (Chapter 3's rule, now enforced by construction).
Report mean and std together. 90.33% alone hides a 0.80–1.00 fold range; the ± 6.57% is half the finding.
Score what matters. scoring='accuracy' was right for this balanced-enough task; Chapter 18's imbalance stories (churn's 80/20!) needed recall, F1, or cost. The metric is a hyperparameter of the question.
Beware the plateau and the noise. A flat top (this chapter's six-way tie) means "any of these"; a huge grid evaluated on small data will crown a noise-champion — Chapter 6's overfitting, relocated from models to model selection.
When the estimate itself must be unbiased, go nested. If you also want an honest performance number after tuning, nest the grid search inside an outer CV loop (inner selects, outer scores). The source skips this; production reviews should not.

⚠

Warning — the reused-validation trap: running grid search, disliking the winner, enlarging the grid, rerunning… each pass spends the same folds' information. After enough reruns, the best CV score optimises the folds, not reality. Budget your search like a finite resource, because it is one.

7. What We Tuned All Along

This course turned dials by hand from early on — each time with a one-split measurement, each time with the nagging question this chapter finally answers. The full retrospective:

Chapter	Dial turned by hand	What we saw	What grid search would add
Ch 6	polynomial degree 1–4	degree 4 best in-sample; bias-variance story	degree + alpha, chosen by CV, not by eyeball
Ch 10	model choice (5 regressors)	linear won the one-split benchmark	cross_val_score for every candidate — the honest tournament
Ch 13–14	C, kernel, γ	kernel menu by single splits (0.74–0.93)	this chapter's exact grid, folded
Ch 17	n_estimators, √p vs p	100 trees settled by inspection	n + max_features on CV
Ch 19	k = 5 by elbow + silhouette	two criteria agreed	(unsupervised — silhouette is the CV analogue)
Ch 30	γ for Kernel PCA	0.944 → 1.000 → 0.444 on one test set	γ swept by CV, plateau identified honestly

Notice the pattern in the last column: nothing conceptually new — the same experiments, automated and averaged. Grid search is not a different kind of thinking; it is the course's habit of "change one thing, measure" with the luck averaged out and the bookkeeping done by machine.

💡

Pro Tip — when tuning matters: this chapter's tuned model gained zero test points because the default was already on the plateau. Tuning pays when defaults sit on a cliff (deep nets, boosting) or when the structural choice (kernel, model family) is genuinely contested. Before tuning anything, ask which kind of landscape you are standing on — the grid search will tell you, and that diagnosis is worth as much as the winner.

⚠ Volatile content — Reviewed: October 2026 · Next review: January 2027

8. The Road Ahead

How the tuning profession has moved beyond exhaustive grids — the fast-moving part, hence the quarterly review:

Randomised search over distributions (not enumerated values) reaches good regions of high-dimensional grids in a fraction of the fits — sklearn's RandomizedSearchCV, often the first upgrade.
Bayesian and bandit-style optimisation (Optuna, Hyperband, successive halving) models which regions look promising and spends fits there — the multi-armed-bandit logic of Chapters 23–24, applied to hyperparameters.
AutoML wraps search space, budget and ensembling into one service — powerful, and best deployed by people who understand exactly what this chapter's rules protect.
The coming chapter needs none of this: XGBoost's brilliance is not exotic tuning but a decade of engineering — gradient-boosted trees (Chapter 17's other ensemble), regularised and made fast. Its parameter grid is exactly where this chapter's discipline gets its workout.

🛠 Hands-On: The Honest TournamentIntermediate

Your team argues about one model's reported accuracy for the umpteenth time. Settle it: quantify the split luck, establish the honest CV number, tune properly, and write the three-sentence verdict nobody can argue with.

Steps

Reproduce Figure 31.1: the same pipeline across ten seeds (0–9), reporting the full range and spread of holdout accuracies.
Hand-verify the fold table: sum the correct-counts to 271/300 = 0.9033, and recompute the std to 0.0657 from the deviations.
Run 10-fold and 5-fold and LOOCV on the same model; confirm the three means and explain why they agree here.
Rerun the CV with a Pipeline (scaler inside); confirm 90.33% ± 6.57% and explain when this distinction would matter.
Run the source's 40-candidate grid search; identify the six-way tie yourself from cv_results_, and the linear ceiling near 0.82.
Evaluate the tuned model once on the test set; compare with the default's 0.930 and write the honest verdict.
Bonus: re-run the grid with scoring='f1' and report whether the winner changes.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import (train_test_split, cross_val_score,
                                     GridSearchCV, LeaveOneOut)
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.metrics import accuracy_score

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values

# ---- step 1: the split lottery
holdout = []
for seed in range(10):
    a_tr, a_te, b_tr, b_te = train_test_split(X, y, test_size=.25,
                                              random_state=seed)
    pipe = make_pipeline(StandardScaler(),
                         SVC(kernel='rbf', random_state=0)).fit(a_tr, b_tr)
    holdout.append(pipe.score(a_te, b_te))
print('holdout range: %.3f..%.3f' % (min(holdout), max(holdout)))
# 0.880..0.950

# ---- steps 2-4: folds
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.25, random_state=0)
sc = StandardScaler(); Xs = sc.fit_transform(Xtr)
clf = SVC(kernel='rbf', random_state=0)
a10 = cross_val_score(clf, Xs, ytr, cv=10)
a5  = cross_val_score(clf, Xs, ytr, cv=5)
alo = cross_val_score(clf, Xs, ytr, cv=LeaveOneOut())
print('10-fold %.4f ± %.4f' % (a10.mean(), a10.std()))   # .9033 ± .0657
print(' 5-fold %.4f ± %.4f' % (a5.mean(),  a5.std()))    # .9033 ± .0386
print(' LOOCV %.4f (%d fits)' % (alo.mean(), len(alo)))  # .9033, 300 fits
pipe = make_pipeline(StandardScaler(), SVC(kernel='rbf', random_state=0))
ap = cross_val_score(pipe, Xtr, ytr, cv=10)
print('pipeline CV %.4f ± %.4f' % (ap.mean(), ap.std())) # same here

# ---- steps 5-6: the grid
parameters = [{'C': [0.25, 0.5, 0.75, 1], 'kernel': ['linear']},
              {'C': [0.25, 0.5, 0.75, 1], 'kernel': ['rbf'],
               'gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]}]
gs = GridSearchCV(SVC(kernel='rbf', random_state=0), parameters,
                  scoring='accuracy', cv=10, n_jobs=-1).fit(Xs, ytr)
print(gs.best_score_, gs.best_params_)     # .9067 {C: .5, gamma: .6, rbf}
res = pd.DataFrame(gs.cv_results_)
top = res[res.mean_test_score == gs.best_score_]
print('tie size:', len(top))               # 6
print('tuned test acc: %.4f' % gs.score(sc.transform(Xte), yte))  # .93

# ---- step 7: F1 scoring
from sklearn.metrics import make_scorer, f1_score
gs_f1 = GridSearchCV(SVC(kernel='rbf', random_state=0), parameters,
                     scoring='f1', cv=10, n_jobs=-1).fit(Xs, ytr)
print('F1 winner:', gs_f1.best_score_, gs_f1.best_params_)

Expected results (real): holdout range 0.880–0.950 (seeds 0–9); pooled folds 271/300 = 0.9033, std 0.0657 (hand-verified); 5-fold 0.9033 ± 0.0386; LOOCV 0.9033 in 300 fits; pipeline CV identical on this data; grid best 0.9067 at {C: 0.5, γ: 0.6, rbf} in a six-way tie; best linear 0.8233; tuned test accuracy 0.930 — identical to the default. Verdict sketch: "The default RBF SVM sits on a wide performance plateau (CV 89–91% across all RBF settings); tuning confirmed rather than improved it. Linear kernels are structurally worse (82%). Reported performance: 90.3% ± 6.6% CV, 93% on the untouched test set."

🎯 Key Takeaways

One split is a lottery: the same SVM returned 0.880–0.950 across ten random splits — a 7-point spread with the model unchanged.

k-fold averages the lottery away: ten stratified folds, every row held out once — pooled 271/300 = 90.33%, the same number as the mean.

Report the pair, not the point: 90.33% ± 6.57% (hand-verified std). Fold range 0.80–1.00 — the ± is half the finding.

Grid search = CV per candidate: 40 candidates × 10 folds = 400 fits. Result: a six-way tie at 0.9067 — plateaus, not needles.

Structure beats fine-tuning: every RBF clears 89%, every linear stalls at 82%. And the tuned model scored 0.930 on test — exactly the default. Confirmation, not transformation.

The six rules: test once, preprocess inside folds, report ±, score the business metric, respect the plateau, go nested when the estimate must be unbiased.