Part III — Classification · Final Chapter

Chapter 18 · Evaluating Classifiers: Beyond Accuracy

Part III built seven classifiers and graded them all with one blunt number. This chapter builds the real judge. Precision, recall, specificity and F1 computed by hand from real confusion matrices; ROC curves constructed threshold by threshold; AUC as ranking skill; thresholds chosen by business costs instead of habit — ending in the grand benchmark, where three different metrics crown three different champions.

📖 Chapter 18 of 33 ⏱ ~95 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Demonstrate exactly how accuracy misleads under class imbalance — with the 99%-accurate useless model
Compute precision, recall, specificity and F1 by hand from a real confusion matrix, and match each to a business error cost
Build a ROC curve threshold by threshold from real data, and read AUC as ranking probability
Choose a threshold by expected cost — Chapter 11's teaser, finally paid off
Read the full eight-model benchmark and defend model choice per business question
Explain calibration as the third evaluation axis, connecting Chapters 15–17's probability caveats

1. Accuracy's Monopoly, Broken

The classic indictment, in one paragraph of arithmetic. A bank screens 10,000 transactions; 100 are fraudulent (1%). A lazy model predicts "legitimate" for every single one. Scorecard: 9,900 correct out of 10,000 — 99% accuracy. Fraud caught: zero. Recall: zero. Value: zero. The model is a parrot with a certificate.

Chapter 11 flagged our own floor (predict-"no"-always scores 64% here), and every chapter since has quietly whispered the same doubt each time an accuracy number appeared. The whisper becomes doctrine now: accuracy is one question — "what fraction of all predictions were right?" — and the two error types it blends may have wildly different costs. A missed buyer costs a margin; a wasted ad costs pocket change. Any single-number metric that cannot tell those apart is not a judge; it is a spectator.

📝

Note: this chapter reuses the Part III bench exactly — 400 users, 300/100 split, seed 0, scaled features — so every number below connects to a chapter you have already read. All eight models (logistic, K-NN, linear SVM, RBF SVM, NB, full tree, depth-2 tree, forest) face the same three new judges: the metric suite, the ROC curve, and the cost table.

2. The Confusion Matrix, Fluently

Every metric in this chapter is a recipe over four counts. With "buyer" as the positive class, the 100 test rows split:

	Predicted
Actual	No	Yes	Row total
No (68)	TN — correctly ignored	FP — false alarm, wasted ad	N = 68
Yes (32)	FN — missed buyer	TP — captured sale	P = 32

Two things to internalise before the formulas arrive. First, which cell hurts most is a business decision, not a statistical one — the entire rest of the chapter is machinery for acting on that decision. Second, the class counts (68/32) live in the matrix's margins: every model faces the same 32 buyers and 68 non-buyers, and the metrics to come will soon disagree about which model served them best.

3. Precision, Recall, Specificity, F1

Four metrics, each answering one stakeholder question:

Metric	Formula	Question it answers	Whose cost it tracks
Precision	TP / (TP + FP)	"When we flag, how often are we right?"	Wasted effort (FP)
Recall (sensitivity)	TP / (TP + FN)	"Of all true buyers, how many did we catch?"	Missed opportunity (FN)
Specificity	TN / (TN + FP)	"Of all non-buyers, how many did we leave alone?"	Harassment of innocents (FP)
F1	2·P·R / (P + R)	"One number balancing precision and recall"	Both, harmonic-mean style

3.1 All four, by hand — logistic's real matrix

Logistic regression (Ch 11): [[65, 3], [8, 24]] — TN = 65, FP = 3, FN = 8, TP = 24.

Precision = 24 / (24 + 3) = 24/27 = 0.889
Recall = 24 / (24 + 8) = 24/32 = 0.750
Specificity = 65 / (65 + 3) = 65/68 = 0.956
F1 = 2 × (0.889 × 0.750) / (0.889 + 0.750) = 2 × 0.667 / 1.639 = 0.814

Verified against sklearn's report to the third decimal. Read the story: flags are trustworthy (89%), but a quarter of all buyers slip through (recall 0.750) — logistic leaves money on the table to protect precision.

Why the harmonic mean for F1? Because it punishes imbalance the ordinary average excuses: precision 1.0 with recall 0.0 averages 0.5 but scores F1 = 0 — you cannot game F1 by maximising one side. (When the costs are asymmetric, the F_β family tilts the balance — F₂ weights recall double.)

3.2 The full suite, all eight models (real)

Model (Ch)	Accuracy	Precision	Recall	F1	AUC
Logistic (11)	0.890	0.889	0.750	0.814	0.954
K-NN (12)	0.930	0.879	0.906	0.892	0.963
SVM linear (13)	0.900	0.923	0.750	0.828	0.953
SVM RBF (14)	0.930	0.879	0.906	0.892	0.965
NB (15)	0.900	0.893	0.781	0.833	0.963
DT full (16)	0.910	0.829	0.906	0.866	0.909
DT depth-2 (16)	0.940	0.882	0.938	0.909	0.943
RF 100 (17)	0.910	0.848	0.875	0.862	0.979

Read the table's boldest lesson before any formula: three different metrics crown three different champions. The depth-2 tree owns accuracy, recall and F1 (0.940 / 0.938 / 0.909). The linear SVM owns precision (0.923 — its wide conservative street flags few, flags right). And the forest — merely fifth on accuracy — owns AUC 0.979, the best ranking of any model: its graded vote shares (Ch 17) order customers by buying likelihood almost perfectly, even though its fixed 0.5-threshold decisions are unremarkable. "Which model is best?" was never a complete question. "Best at what" is the question this table forces you to ask.

💡

Pro Tip — the metric before the model: screening for a rare disease? Recall (missing a case is catastrophic). Approving loans? Precision (a bad approval costs the principal). Ranking a marketing database top-to-bottom? AUC. Decide the metric with the business, then let the benchmark speak.

4. ROC Curves, Built by Hand

A model with probabilities gives not one classifier but a whole family: every threshold t is a different operating point, trading recall against precision. The ROC curve traces that family — true-positive rate against false-positive rate as t sweeps from 1 down to 0. Real points from logistic regression (Ch 11's model, probabilities and all):

Threshold	0.9	0.8	0.7	0.6	0.5	0.4	0.3	0.2	0.1
TPR (recall)	0.344	0.531	0.656	0.656	0.750	0.844	0.906	0.969	0.969
FPR	0.015	0.015	0.015	0.015	0.044	0.103	0.147	0.250	0.368
FP / FN	1/21	1/15	1/11	1/11	3/8	7/5	10/3	17/1	25/1

Walk one row of that table in slow motion. At t = 0.3: 29 of 32 buyers caught (TPR 0.906) at the price of 10 false alarms (FPR 0.147). Slide to t = 0.5: recall drops to 0.750 but false alarms collapse to 3. The curve is the trade-off, and no single point on it is "the model."

Figure 18.1 — Logistic's real ROC: each dot an operating threshold; the shaded-free area under the curve is AUC = 0.954. Better models bow harder toward the top-left.

4.1 AUC — the area as a probability

AUC (area under the ROC curve) has the most useful interpretation in all of evaluation: the probability that the model ranks a random buyer above a random non-buyer. 1.0 = perfect ordering; 0.5 = coin-flip ranking (the diagonal); below 0.5 = worse than random — flip your predictions. And it explains the benchmark's surprise: the forest's vote shares (Ch 17's [0.03, 0.97]-style graded outputs) order the 100 customers almost perfectly (AUC 0.979) — ranking talent its 0.5-threshold accuracy never revealed. AUC is threshold-free: it judges the probabilities, not one arbitrary cut.

⚠

Warning — ROC vs PR for rare classes: with 1% positives, the FPR axis flatters (tiny FP counts look great against a huge negative class). The precision–recall curve is the honest picture for rare-disease and fraud problems. Moderate imbalance like ours (32/68)? ROC tells the story cleanly.

5. Cost-Based Thresholds

Chapter 11 deferred the question; the machinery now exists. Assign each error a price — say a missed buyer costs ₹10,000 (lost margin) and a wasted ad costs ₹500. For every threshold, the expected cost is FN × 10,000 + FP × 500 — all counts straight from the ROC table's source. The real cost curve (logistic):

Threshold	0.05	0.10	0.25	0.30	0.45	0.50 (default)	0.60	0.75	0.90
FP / FN	35/0	25/1	15/1	10/3	4/6	3/8	1/11	1/14	1/21
Cost (₹)	17,500	22,500	17,500	35,000	62,000	81,500	110,500	140,500	210,500

The verdict is brutal for habit. The default 0.5 threshold — the one every tutorial ships — costs ₹81,500 across this test set. Sliding down to 0.05 (or the tied 0.25) costs ₹17,500: catching every single buyer while tolerating 35 cheap false ads. Under these prices, missing a buyer is twenty times worse than chasing a ghost, and the optimal operating point sits far from 0.5. Change the prices and the optimum moves — that is not a weakness of the method; it is the method.

🌎

Real World: this is precisely how fraud, churn and disease-screening systems set thresholds in production — an explicit cost matrix (FN × cost-of-miss + FP × cost-of-alarm), minimised over the validation set, revisited whenever the business costs change. The model ranks; the costs decide.

6. The Grand Benchmark

All eight models, three judges, one table (Section 3.2's numbers, now with their verdicts):

Business question	Winning model (real)	Winning numbers	Why
"One hard yes/no decision per customer"	DT depth-2	acc 0.940 · F1 0.909 · recall 0.938	The boxy truth fits three rules; misses only 2 of 32 buyers
"Rank the database, marketing picks the top-N"	Forest	AUC 0.979	Graded vote shares order buyers above non-buyers almost perfectly — Ch 17's calibration dividend
"Every flag must be defensible"	SVM linear	precision 0.923	The wide conservative street: flags 26, wrong only 2
"Catch every buyer, ads are cheap"	Any model + threshold 0.05–0.25	recall ≈ 1.0 at cost ₹17,500	Section 5: the threshold, not the model, owns this question

The recommendation template from Chapter 10, now in classification dress: name the decision type, name the metric, then name the model. And note what the table quietly demonstrates: the "best model" question was always under-specified. Seven chapters of single-number scoreboards hid that the forest was the best ranker, the stump-tree the best decider, and the SVM the best skeptic — simultaneously, on the same 100 rows.

💡

Pro Tip: the small-sample caveat, one last time — these gaps ride on 100 test rows (one buyer = 3 recall points), and every sweep in this chapter consulted the test set. The deployable protocol — tune thresholds inside cross-validation, audit the final choice once — is Chapter 31's. The structure of the conclusions, not their exact decimals, is what survives.

📋 Stable content — Reviewed: August 2026

7. Implementation in Python

from sklearn.metrics import (classification_report, confusion_matrix,
                             accuracy_score, precision_score, recall_score,
                             f1_score, roc_auc_score, roc_curve)

y_pred = model.predict(X_test)
probs  = model.predict_proba(X_test)[:, 1]      # P(buyer) per row

# The whole suite in one call
print(classification_report(y_test, y_pred,
                            target_names=['No', 'Yes']))
#           precision  recall  f1-score  support
# No             ...      ...      ...       68
# Yes            ...      ...      ...       32
# accuracy                        ...      100
# macro avg      ...      ...      ...
# weighted avg   ...      ...      ...       # weighted by class size

# Or metric by metric
print(precision_score(y_test, y_pred))    # 0.889 (logistic)
print(recall_score(y_test, y_pred))       # 0.750
print(f1_score(y_test, y_pred))           # 0.814
print(roc_auc_score(y_test, probs))       # 0.954

# The ROC machinery, for plotting or thresholds
fpr, tpr, thresholds = roc_curve(y_test, probs)

# The cost-optimal threshold, straight from Section 5
FP_COST, FN_COST = 500, 10_000
costs = [FN_COST*((probs < t) & (y_test == 1)).sum() +
         FP_COST*((probs >= t) & (y_test == 0)).sum()
         for t in thresholds]
best = thresholds[int(np.argmin(costs))]
print(f'cost-optimal threshold: {best:.2f}')

Listing 18.1 — The evaluation suite: report, metrics, ROC, cost-optimal threshold.

# The one-call visuals
from sklearn.metrics import ConfusionMatrixDisplay, RocCurveDisplay
import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4))
ConfusionMatrixDisplay.from_estimator(model, X_test, y_test, ax=ax1)
RocCurveDisplay.from_estimator(model, X_test, y_test, ax=ax2)
plt.show()

# Compare several models' ROCs on one canvas
for name, m in models.items():
    RocCurveDisplay.from_estimator(m, X_test, y_test, ax=ax2, name=name)

Listing 18.2 — Modern one-call displays replace the source's hand-rolled plots.

💡

Pro Tip — probabilities from SVMs: SVC offers decision_function (distances) natively — enough for ROC/AUC. For real probabilities, current sklearn prefers wrapping the model: CalibratedClassifierCV(SVC(), ensemble=False); the old probability=True flag still works but is deprecated. AUC needs only a ranking, so distances serve perfectly.

8. Calibration & the Road Ahead

This chapter's third axis, hinted across three chapters, now has a name: calibration — when a model says 0.90, do 90% of such cases actually happen? AUC only needs correct ranking; costs need correct ranking; but a business acting on the number itself ("pre-approve everyone above 0.8") needs calibrated probabilities. The cast's honesty audit, condensed: logistic and NB output genuine probability-shaped objects (NB over-committed at the tails, Ch 15); tree leaves over-commit catastrophically ([0, 1] from a handful of rows, Ch 16); the forest's averaged votes land in between — excellent ranking (AUC 0.979), graded but not guaranteed calibrated. The repair kit — reliability diagrams, Platt scaling, isotonic regression — wraps any ranker into a calibrated one, and lives in CalibratedClassifierCV.

With the judge complete, the supervised journey — both Parts — is finished: thirteen models, each measured now the honest way. Part IV removes the last scaffolding: the labels themselves. Clustering asks what structure survives when nobody tells you the answer — and its first lesson is that even "how good is this model?" loses its confusion matrix, replaced by a hunt for internal measures of shape.

🛠 Hands-On: Judge the FieldIntermediate

The CMO asks three different questions in one meeting: "Who should we definitely target?", "Rank our database for the campaign", and "What's the cheapest way to run this?" Produce the three answers — metrics by hand, ROC, cost curve — from the model bench.

Steps

By hand: from DT depth-2's matrix [[64, 4], [2, 30]], compute precision, recall, specificity and F1; verify against sklearn.
Produce the full eight-model metric table (accuracy, precision, recall, F1, AUC); confirm the three different champions.
Extract logistic's probabilities; build the threshold table (TPR/FPR at 0.1…0.9) and sketch or plot the ROC; verify AUC 0.954.
Under costs FN = ₹10,000 / FP = ₹500, compute the cost curve and find the optimum; then reverse the costs (FN = ₹500 / FP = ₹10,000) and find the new optimum.
One-plot ROC comparison of all eight models; describe which curve hugs the corner hardest.
Write the three answers for the CMO — decision, ranking, cheapest — each with its model, metric and number.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (classification_report, roc_auc_score,
                             roc_curve, precision_score, recall_score,
                             f1_score)

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
lr = LogisticRegression(random_state=0).fit(sc.fit_transform(X_tr), y_tr)
probs = lr.predict_proba(sc.transform(X_te))[:, 1]

# Step 1 (hand check for DT-2)
P, R = 30/34, 30/32
print('DT-2 hand: P=%.3f R=%.3f F1=%.3f' % (P, R, 2*P*R/(P+R)))

# Steps 3-4
fpr, tpr, thr = roc_curve(y_te, probs)
print('AUC %.3f' % roc_auc_score(y_te, probs))
for FNc, FPc in [(10_000, 500), (500, 10_000)]:
    costs = [FNc*((probs < t) & (y_te == 1)).sum() +
             FPc*((probs >= t) & (y_te == 0)).sum() for t in thr]
    i = int(np.argmin(costs))
    print('costs FN=%d FP=%d -> best t=%.2f (₹%d)' % (FNc, FPc, thr[i], costs[i]))

Expected results (real):

Step 1: DT-2: P = 30/34 = 0.882, R = 30/32 = 0.938, spec = 64/68 = 0.941, F1 = 0.909 — matches sklearn to the third decimal.
Step 2: champions — DT-2 (acc/recall/F1: 0.940/0.938/0.909), SVM linear (precision 0.923), forest (AUC 0.979).
Steps 3–4: logistic AUC 0.954; cost curve minimum ₹17,500 at t ≈ 0.05 (tied at 0.25) under 10,000/500; with costs reversed the optimum leaps to the high end (t ≈ 0.6+: 1 FP dominates) — the threshold follows the money.
Step 5: the forest's curve hugs hardest (AUC 0.979); DT full's is visibly kinked (AUC 0.909 — coarse leaf probabilities rank poorly).
Step 6: decision → DT-2 (0.940); ranking → forest (AUC 0.979); cheapest → logistic (or any) at the cost-optimal threshold — ₹17,500 vs ₹81,500 at default.

🎯 Key Takeaways

Accuracy blends two error types that may cost wildly different amounts — under imbalance it certifies parrots (99% accurate, zero fraud caught).

Precision guards wasted effort, recall guards missed opportunity, F1 refuses to let you game one side — all computable by hand from four counts.

A threshold is an operating point on a ROC curve, not "the model". AUC = the probability a random buyer outscores a random non-buyer.

Three metrics crowned three champions: DT-2 decides best (F1 0.909), SVM linear flags most defensibly (P 0.923), the forest ranks best (AUC 0.979).

Thresholds belong to the business: under real costs the optimum moved from 0.5 (₹81,500) to 0.05–0.25 (₹17,500) — a 4.7× saving from one dial.

Calibration is the third axis: ranking skill (AUC) ≠ honest probabilities. When the business acts on the number, calibrate first.