Chapter 10 · Evaluating & Comparing Regression Models
Part II built six regressors; this chapter builds the judge. Error metrics computed by hand and read in business units, the residual audit that certifies validity, and cross-validation that replaces one lucky split with five honest ones — all concluding in a real benchmark where every model faces the same data, and the results overturn expectations.
🎯 Learning Objectives
- Compute MAE, MSE, RMSE and RMSLE by hand, and state precisely when each is the right lens
- Explain why in-sample R² crowns memorisers — and what test R² and adjusted R² repair
- Run the four-plot residual audit that certifies (or convicts) any regression
- Apply k-fold cross-validation to regression, report mean ± SD, and avoid per-fold leakage with pipelines
- Read a real six-model benchmark table and defend a recommendation in writing
- Choose evaluation metrics from business costs, not habit
1. The Judgement Layer
Two chapters ago a decision tree "won" R² = 1.0 on the salary grid; last chapter a forest scored a humbler 0.97 and we called it healthier. This chapter makes that judgement systematic. Evaluation is not a report at the end of modelling — it is the specification of what "better" means, written before the models compete.
Three instruments, each answering a different question, none sufficient alone:
- Error metrics (Sections 2–3) — how much do we miss, in the target's own units?
- Variance-share metrics (Section 4) — what fraction of the target's variation does the model capture?
- Diagnostics & validation design (Sections 5–6) — can the model be trusted, and on which data was it measured?
Salary_Data.csv (30 employees, experience →
salary) with the Chapter 4 split (1/3 test, seed 0), so every number you are
about to see connects back to a chapter you have already read.
2. The Metric Toolkit: MAE, MSE, RMSE
Four residuals from a fitted model: e = 3, −4, 2, −1 (salary misses in ₹000s). Every metric below is computed from exactly these four numbers:
- MAE — the typical miss, in salary units. Every rupee of error counts one rupee.
- MSE — squared units (₹000s²): mathematically convenient (differentiable — the loss OLS and gradient descent actually minimise) but not speakable to a manager.
- RMSE — back in salary units, but still carrying the squaring's attitude: large misses count disproportionately.
That attitude is the whole MAE-vs-RMSE story. Watch the same trick with residuals 2, 2, 2, 2, 10 (real arithmetic):
One outlier among five ordinary errors: MAE rises meekly, RMSE screams. Neither is "correct" — they are two cost functions. If a ₹20,000 miss is just four times worse than a ₹5,000 miss, your metric is MAE. If it is catastrophically worse — a missed delivery prediction breaks a contract, a missed fraud threshold costs the transaction — your metric is RMSE. The business decides the metric; the metric then decides the model.
3. RMSLE & When Metrics Disagree
Some errors should be judged relatively. A ₹10 miss on a ₹20 salary prediction is disastrous; the same ₹10 miss on ₹1,00,000 is noise. RMSLE scores errors in log space, making them proportional:
When to reach for which — a decision table, not a ranking:
| Metric | Units | Attitude to outliers | Reach for it when… |
|---|---|---|---|
| MAE | target units | linear — each rupee counts once | you want the honest "typical miss"; errors are uniformly costly |
| MSE | squared units | harsh | optimisation and math (it is the loss being minimised); comparing models on the same data |
| RMSE | target units | harsh, but speakable | large misses cost disproportionately — the default business report |
| RMSLE | log-relative | punishes relative misses | targets span orders of magnitude (demand 10→10,000; prices; salaries across bands) |
| R² | share of variance | implicit | communicating "how much of the variation is explained" — never alone (Section 4) |
4. R² and Adjusted R², Honestly
Chapter 4 derived R² = 1 − SSE/SST: the share of variation explained. Three chapters of experience now let us state its rules of honest use:
- Which set? R² must be computed on held-out data to mean anything about generalisation. In-sample R² mechanically rewards complexity — the memorising tree's 1.0 was this trap in full costume. Test R², by contrast, is a genuine out-of-sample variance share.
- Which units? R² carries none. A model can explain 97% of variance and still miss by ₹37,000 (Section 7 shows exactly this). Pair every R² with an RMSE: share and size.
- How many features? On training data, adding features never lowers R² (Chapter 5); adjusted R² charges the rent: 1 − (1−R²)(n−1)/(n−p−1). Use it when comparing models of different sizes in-sample; on a test set, plain R² already contains the honesty.
5. The Residual Audit Checklist
Metrics score a model; diagnostics certify it. The audit from Chapter 4 generalises to every regressor in Part II (tree steps, kernel bumps and ensemble averages all produce residuals), and it takes four plots:
| # | Plot | Assumption on trial | Convicted by | Standard response |
|---|---|---|---|---|
| 1 | Residuals vs fitted | Linearity & equal variance | Curve (non-linearity); funnel (heteroscedasticity) | Transform (log y); richer model family |
| 2 | Q–Q plot of residuals | Normality of errors | Tails bending off the diagonal | Transform y; collect more data; robust intervals |
| 3 | Residuals vs observation order | Independence | Waves or runs (autocorrelation) | Time-series structure; add trend/season features |
| 4 | Residual magnitude vs leverage / outlier fences | No single point steering the fit | Extreme Cook's distance; |z| > 3 residuals | Investigate the row; correct or justify — never silently delete |
# The audit, four lines at a time
residuals = y_test - model.predict(X_test)
plt.scatter(model.predict(X_test), residuals); plt.axhline(0, ls='--', c='grey')
plt.xlabel('Fitted'); plt.ylabel('Residual'); plt.title('1·3 audit: curve? funnel?')
stats.probplot(residuals, plot=plt) # 2: normality
plt.plot(residuals, marker='o') # 3: order pattern?Listing 10.1 — The audit runs on test residuals where available; training residuals when data is scarce.
The audit's verdict is qualitative by design — it certifies that the quantitative metrics above are measuring something real. A benchmark winner with a funnel-shaped residual plot has not won; it has merely averaged its pathology politely.
6. Cross-Validation for Regression
Chapter 4's test set held ten rows; a lucky or unlucky draw moves a small-set score by points. k-fold cross-validation spends the same data five times smarter: shuffle, split into k = 5 folds, train on four, test on one, rotate — every row is tested exactly once, on a model that never saw it:
from sklearn.model_selection import cross_val_score, KFold
from sklearn.pipeline import Pipeline
cv = KFold(n_splits=5, shuffle=True, random_state=42)
# Pipeline = leakage-proof CV: scalers refit inside every fold
model = Pipeline([('sx', StandardScaler()),
('m', SVR(kernel='rbf', C=10))])
scores = cross_val_score(model, X, y, cv=cv, scoring='r2')
print(f'{scores.mean():.3f} ± {scores.std():.3f}') # report BOTH numbersListing 10.2 — The CV loop. Without the pipeline, the scaler fitted on all folds' data would leak — Chapter 3's rule, now per fold.
The report is mean ± SD: the mean estimates performance, the SD estimates how much that estimate wobbles. A model at 0.92 ± 0.05 and one at 0.90 ± 0.02 are not ranked by their means alone — the second is more predictable, which production teams value. Full treatment of tuning with CV is Chapter 31; here CV is the benchmark's honesty device.
7. The Benchmark: Six Models, One Dataset
The rules: Salary_Data.csv (30 employees), the Chapter 4 split
(20 train / 10 test, seed 0), one shared metric suite, no per-model tuning beyond
the configurations earlier chapters established. Every number below is real.
7.1 Held-out test results
| Model (Ch) | MAE (₹) | RMSE (₹) | R² test | R² train | Gap (train − test) |
|---|---|---|---|---|---|
| Linear (4) | 3,426 | 4,585 | 0.9749 | 0.938 | −0.037 |
| Poly deg-2 (6) | 3,563 | 4,643 | 0.9743 | 0.939 | −0.035 |
| Poly deg-4 (6) | 4,215 | 4,705 | 0.9736 | 0.950 | −0.024 |
| SVR default (7) | 30,727 | 37,087 | −0.641 | −0.114 | +0.53 |
| SVR C=10 (7) | 30,689 | 37,040 | −0.637 | −0.110 | +0.53 |
| Tree (8) | 6,840 | 8,685 | 0.910 | 0.996 | +0.086 |
| Forest, 100 (9) | 5,355 | 6,638 | 0.947 | 0.986 | +0.039 |
Five readings, each one a chapter's lesson made numerical:
- The simplest model wins outright. Untuned linear regression posts the best MAE, the best RMSE and the best test R² of the entire field. The data is nearly linear (r = 0.978, Chapter 4) — and the benchmark simply confirms it: flexibility bought nothing here. Occam's razor, audited.
- The memorisation gap is visible per model. The tree: train 0.996 → test 0.910 (gap +0.086). The forest halves its own tree's gap (+0.039) — Chapter 9's variance-cancellation, measured. The linear model's gap is negative (test > train) — the harmless small-sample luck of Chapter 4.
- Negative R² is real. Both SVR configurations score −0.64 on test — worse than predicting the training mean. Chapter 7's warning — "never apply settings as slogans" — here in the flesh: the tube-γ-ε configuration that delighted us on ten tidy levels fails on thirty noisier salaries.
- MAE–RMSE agreement as outlier check. For the linear model, RMSE 4,585 vs MAE 3,426 — same order, no dominant outlier. For the failed SVR, both are ~₹30–37k: uniformly bad, not sporadically bad — a systematic misfit, not a few bad rows.
- MLR's cameo (Chapter 5): on its natural home (50_Startups), multiple regression posted test R² 0.947, RMSE ₹8,199 — competitive everywhere, champion nowhere, indispensable when the question is which features matter.
7.2 Cross-validated stability (5 folds, all 30 rows)
| Model | Linear | Poly d2 | Poly d4 | SVR C=10 | Tree | Forest 100 |
|---|---|---|---|---|---|---|
| CV R² (mean ± SD) | 0.922 ± 0.050 | 0.914 ± 0.052 | 0.924 ± 0.051 | −0.61 ± 0.55 | 0.878 ± 0.088 | 0.917 ± 0.068 |
CV confirms the test table and adds nuance: the linear and degree-4 models are statistically inseparable (0.92 ± 0.05 both) — a tie broken, correctly, by simplicity. The tree is the wobbliest winner-adjacent model (± 0.088); the forest, as promised, sits between. And SVR's ± 0.55 says it all: not just bad but erratically bad.
7.3 Can SVR be rescued? Yes — at a price
A grid over C, γ and ε (108 configurations) finds C=10, γ=0.05, ε=0.01 with test R² 0.9794 — the best single score in this whole chapter. Two honest caveats attach to that number. First, it took 108 attempts to beat the linear model by 0.005 — effort the linear model spent on nothing. Second — the deeper lesson — the grid was scored on the test set, so 0.9794 is an optimised, optimistic statistic: the test set has now been consulted 108 times, and Chapter 3's "touch it once" rule lies in pieces. The correct protocol — tune inside cross-validation on training data only, then one final test evaluation — is exactly Chapter 31's subject. The benchmark keeps its honest winner.
8. Choosing a Model: The Decision Guide
The benchmark's deepest result is procedural, not numerical. Models are chosen by the question being asked, under the evidence the metrics and audits provide:
| If the priority is… | Reach for | Because |
|---|---|---|
| Explaining to humans ("which factors, how much") | Linear / MLR (Ch 4–5) | Coefficients in business units; inference (p-values, VIF); the audit suite applies cleanly |
| Gentle, visible curvature | Poly deg 2–3 (Ch 6) | Transparent equation that bends; keep degree low, ban extrapolation |
| Nonlinear, noisy, medium-size data | SVR (Ch 7) | Tube noise-immunity + kernel flexibility — after systematic tuning |
| Auditability by non-experts (rules, compliance) | Tree (Ch 8) | A flowchart you can read aloud; control depth against memorisation |
| Best out-of-the-box accuracy on tabular data | Forest (Ch 9) | Variance-cancelling default; OOB gives a free honesty check |
| Maximum accuracy, tuning budget available | Boosting (Ch 32 preview) | Sequential residual-chasing; the other ensemble family |
And the reporting template, now complete — reuse it for the rest of this course:
Part II ends here — six builders and one judge. Part III changes the question: not "how much?" but "which one?" — predicting categories instead of numbers, starting with logistic regression, where a regression's skeleton learns to classify.
🛠 Hands-On: Run Your Own BenchmarkIntermediate
A rival team claims forests dominate everything. Run the full benchmark on
Salary_Data.csv yourself — metrics, gaps, CV — audit the winner's
residuals, and deliver the five-part recommendation.
Steps
- Split 1/3 test, seed 0. Fit linear, poly-2, poly-4, SVR (default and C=10), tree, forest-100. One shared metrics function.
- Produce the table: MAE, RMSE, R² test, R² train, and the train−test gap per model.
- Which model wins on RMSE? Compute the MAE–RMSE ratio for the winner and interpret it (outlier check).
- Run 5-fold CV (shuffle, seed 42) for all models — in pipelines where scaling is involved. Report mean ± SD.
- Audit the winner's test residuals (four plots). Verdict?
- Write the five-part recommendation. Would a tuned SVR change it — and what protocol would that tuning need?
View Solution / Walkthrough
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score, KFold
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
ds = pd.read_csv('Salary_Data.csv')
X = ds.iloc[:, :-1].values.astype(float)
y = ds.iloc[:, -1].values.astype(float)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=1/3, random_state=0)
models = {
'Linear' : LinearRegression(),
'Poly d2' : Pipeline([('p', PolynomialFeatures(2, include_bias=False)), ('m', LinearRegression())]),
'Poly d4' : Pipeline([('p', PolynomialFeatures(4, include_bias=False)), ('m', LinearRegression())]),
'SVR' : Pipeline([('s', StandardScaler()), ('m', SVR(kernel='rbf'))]),
'SVR C=10': Pipeline([('s', StandardScaler()), ('m', SVR(kernel='rbf', C=10))]),
'Tree' : DecisionTreeRegressor(random_state=0),
'Forest' : RandomForestRegressor(n_estimators=100, random_state=0),
}
for name, m in models.items():
m.fit(X_tr, y_tr)
yp = m.predict(X_te)
print('%-9s MAE=%6.0f RMSE=%6.0f R2test=%7.4f R2train=%7.4f' %
(name, mean_absolute_error(y_te, yp),
np.sqrt(mean_squared_error(y_te, yp)),
r2_score(y_te, yp), m.score(X_tr, y_tr)))
cv = KFold(n_splits=5, shuffle=True, random_state=42)
for name, m in models.items():
s = cross_val_score(m, X, y, cv=cv, scoring='r2')
print('%-9s CV %.3f ± %.3f' % (name, s.mean(), s.std()))Expected results (real — Sections 7.1–7.2):
- Winner: linear — MAE 3,426, RMSE 4,585, R²test 0.9749. The forest is a solid second (RMSE 6,638), but the claim "dominates everything" is refuted by the data's own linearity.
- Gap column: tree +0.086 (memoriser), forest +0.039 (halved), linear −0.037 (small-sample luck, benign).
- MAE–RMSE ratio (winner): 3,426/4,585 ≈ 0.75 — no dominant outlier; the miss profile is evenly spread.
- CV: linear 0.922 ± 0.050 vs poly-4 0.924 ± 0.051 — a statistical tie; SVR −0.61 ± 0.55; tree wobbliest (± 0.088).
- Audit: the linear winner's residuals show no curve, no funnel (salary data is well-behaved), Q–Q near-diagonal — the win is certified, not just scored.
- Recommendation: five parts per the template, landing on linear + forest as the deployment pair (champion + challenger); tuned SVR (0.9794 on this test set) noted as optimistic unless re-run under the Chapter 31 protocol.
🎯 Key Takeaways
Metrics are cost functions: MAE counts every rupee once, RMSE punishes the big miss. The business's error costs choose the metric before any model runs.
RMSE ≥ MAE always — and the gap between them is a free outlier detector for any model's error profile.
R² is a share, not a size: pair it with RMSE, compute it on held-out data, and remember negative test R² means "worse than the mean" — informative, not broken.
Four plots certify any regression: residuals-vs-fitted, Q–Q, vs-order, leverage. A benchmark winner with a funnel has averaged its pathology, not fixed it.
k-fold CV replaces one lucky split with mean ± SD — and pipelines keep the folds leakage-free. Ties between equal scores break toward simplicity.
The real benchmark's verdict: untuned linear regression beat five fancier models, and a tuned SVR's win came from consulting the test set 108 times. Simple baselines are the professional's first move.
Test Your Understanding
30 questions across the chapter — metric computations and choices, honest R², the residual audit, cross-validation, and the real benchmark. Choose one answer per question, then check your answers.
Q1.This chapter exists because, left to in-sample scores:
Q2.MAE is defined as:
Q3.(Hand) Residuals 3, −4, 2, −1. The MAE is:
Q4.(Hand) The same residuals give MSE:
Q5.(Hand) The RMSE of those residuals is ≈ 2.74, which is:
Q6.Residuals 2, 2, 2, 2, 10 give (real) MAE = 3.60 and RMSE ≈ 4.82. The divergence shows:
Q7.Your business treats a ₹20,000 miss as merely four times a ₹5,000 miss. The matching metric is:
Q8.MSE, despite its awkward units, survives everywhere because:
Q9.RMSLE exists for problems where errors should be judged:
Q10.(Real mini-example) Predictions 100→110 and 20→10 — equal absolute misses — score RMSLE 0.46. The reading:
Q11.A missed prediction breaks a contract and costs a fortune, while ordinary misses are routine. Choose:
Q12.A test R² of 0.95 states that:
Q13.In-sample R² "lies" because it:
Q14.Adjusted R² exists to:
Q15.The pairing rule for reporting is:
Q16.The first and most information-dense audit plot is:
Q17.A funnel in the residuals-vs-fitted plot convicts the model of:
Q18.In the audit, the Q–Q plot tests:
Q19.Plotting residuals against observation order is the check for:
Q20.A single 10-row test split is fragile because:
Q21.In k-fold cross-validation:
Q22.The professional CV report is:
Q23.Inside CV, scaling must be wrapped in a Pipeline because:
Q24.(Real benchmark) Among the untuned models on Salary_Data's test set, the winner was:
Q25.(Real) The tree scored R² 0.996 on train but 0.910 on test; the forest 0.986 vs 0.947. The diagnosis:
Q26.(Real) SVR scored test R² ≈ −0.64 on this dataset, yet a 108-config grid found one at 0.9794. The honest reading:
Q27.The stakeholder asks "which features matter, and how much each?". The decision guide says:
Q28.Two models tie on CV (0.922 ± 0.050 vs 0.924 ± 0.051). The professional choice:
Q29.The five-part report template ends with:
Q30.Part III, starting next chapter, changes the question from "how much?" to: