Part X — Model Selection & Boosting

Chapter 33 · Capstone: End-to-End ML Project

Thirty-two chapters built and judged models one at a time. This final chapter builds the profession: a single pipeline from raw business data to a deployed model with a money-denominated case. The client is a bank's retention team with 10,000 customer records and a churn problem; the deliverable is a tuned champion, a call list, an expected-profit figure, and a deployed artifact with a monitoring plan. Every stage is one course chapter wearing its work clothes — Chapter 3's audit, Chapter 31's shortlist and tuning, Chapter 18's business metrics, Chapter 32's champion — and every number on this page, including the $610,800 the model is worth, comes from the real run.

📖 Chapter 33 of 33 ⏱ ~85 min read Mostly Core Stable Volatile 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Translate a business brief into a machine-learning plan: question, data, metric, deliverable
Audit and prepare raw data the professional way: identifiers out, encodings in, 80/20 imbalance on the radar
Run a model shortlist by 10-fold cross-validation with AUC as the ranking metric — logistic, tree, forest, XGBoost — then tune both finalists
Touch the test set exactly once and read the full metric suite, against the majority baseline and Chapter 26's neural network
Convert probabilities into money: a $200 offer, a $2,000 customer, a threshold sweep to the $610,800 optimum at t = 0.09 — and the formula that predicted it
Ship: the deployment artifact, the monitoring plan, and the ten-step workflow that is this course compressed into a checklist

1. The Brief

A realistic client conversation, compressed: "Customers leave. Marketing says replacing one costs five times keeping one. We have 10,000 customer records and a retention budget. Tell us who to call." Chapter 26 already met this dataset through the neural network; the capstone's job is the whole professional response — and its shape is always the same:

Restate as a prediction problem: estimate each customer's churn probability (not a yes/no — the money will set the cutoff, Section 5).
Audit the data: Churn_Modelling.csv — 10,000 rows; drop RowNumber, CustomerId, Surname (Chapter 3's identifier rule); label-encode Gender; one-hot Geography (France/Germany/Spain in front); 12 features; split 8,000 train / 2,000 test, seed 0; standardise on train only.
Face the imbalance now, not later: churners are 20.4% of the data (1,632 train / 405 test). The lazy-classifier floor is 79.75% accuracy — accuracy will flatter everything, so the working metric is chosen with care (Section 2).
Agree the deliverable: a tuned model, a threshold, an expected-profit number, a deployable artifact, and a monitoring plan.

📝

Note — why this dataset: it has identifier columns to drop, categories to encode, imbalance to respect, and a genuine business stake — every preprocessing chapter's lesson in one file. It also lets the capstone compete with Chapter 26's neural network on equal terms: same split, same scaling, same 2,000-customer jury.

2. The Shortlist

Chapter 31's discipline: no more single-split beauty contests. Four families — deliberately spanning the course — each given a 10-fold cross-validation inside the 8,000 training rows, scored on three metrics at once. The headline metric is ROC-AUC, for a reason worth stating aloud: the deliverable is a ranking (who to call first), and AUC measures ranking quality independently of any threshold — the threshold itself is a Section 5 business decision, not a statistical one.

Model family	AUC (10-fold)	Recall	Accuracy	Verdict
Logistic regression (Ch 11)	0.763 ± 0.023	0.210	0.808	baseline; blind to 4 of 5 churners
Decision tree (Ch 16)	0.676 ± 0.025	0.487	0.788	ranking too crude alone
Random forest (Ch 17)	0.851 ± 0.018	0.461	0.861	best untuned ranker
XGBoost (Ch 32)	0.839 ± 0.021	0.490	0.852	close second — but the more tunable

Read it honestly, in Chapter 31's grammar: means and spreads. The two ensembles sit ~0.85 with overlapping uncertainty — the forest nominally ahead untuned. The linear model's AUC of 0.763 confirms Chapter 26's finding (its recall of 0.21 barely beats ignoring churners); the lone tree ranks worst of all — Chapter 17's lesson that trees want company, quantified. Two finalists advance to tuning: the forest and XGBoost.

💡

Pro Tip — shortlists are cheap, so make them: the whole table cost seconds of compute. Ten minutes of CV on five families routinely kills a week of tuning the wrong one — and a shortlist with a linear model in it doubles as the interpretability fallback if the business demands reasons.

3. Tuning the Champion

Both finalists get a small, principled grid — Chapter 31's rules (CV inside train, test untouched, audit the whole table, not just the winner):

XGBoost: n_estimators {100, 300} × max_depth {3, 6} × learning_rate {0.05, 0.2} × subsample {0.8, 1.0} — 16 candidates × 5 folds = 80 fits.
Random forest: n_estimators {100, 300} × max_depth {None, 10} × max_features {√p, 0.5} — 12 candidates × 5 folds = 60 fits.

Finalist, tuned by CV AUC	Best params	CV AUC
XGBoost	lr 0.05 · depth 3 · 300 trees · subsample 0.8	0.8638
Random forest	300 trees · depth 10 · √p features	0.8620

A whisker apart — and that is the finding. Tuning lifted XGBoost from 0.839 to 0.8638 (overtaking the forest's 0.851), but the tuned forest answered with 0.8620; the gap is 0.002, far inside the folds' own ± 0.02 spread. Chapter 31's plateau, one last time: on tabular data with good defaults, tuning confirms rather than transforms. The XGBoost configuration takes the crown on the number, with the honest footnote that either finalist would have shipped. Note also the winning shape — depth 3, learning rate 0.05, 300 small trees — exactly Chapter 32's "many weak, careful steps" recipe.

4. The Test Set — Once

The champion, refit on all 8,000 training rows, now meets the 2,000 held-out customers. One evaluation, full metric suite, no second attempts:

	Confusion matrix (t = 0.5)	Acc	Precision	Recall	F1	AUC
Capstone champion (XGB)	[[1518, 77], [190, 215]]	0.866	0.736	0.531	0.617	0.875
Ch 26 neural network (12-6-6-1)	[[1511, 84], [192, 213]]	0.862	0.717	0.526	0.607	—
Majority ("nobody leaves")	[[1595, 0], [405, 0]]	0.798	—	0.000	—	0.500

The champion edges the neural network on every line — by a handful of the same customers, which Chapter 31 taught us to say out loud. The AUC of 0.875 is the number that matters: it says the model ranks customers well, whatever threshold the business later picks. The probabilities are graded and healthy (min 0.004, mean 0.217 ≈ the churn rate, max 0.992).

One calibration probe, borrowed from Chapter 18: feed the champion the two customers Chapter 26's network judged. The safe customer (France, 40, active): 0.044 (the ANN said 0.029). The risky one (Germany, 65, inactive): 0.579 (the ANN said 0.936). Same ordering, different confidence — trees, built from threshold splits, do not extrapolate beyond their leaves, so extreme predictions come out conservative. For a call-list decision both are actionable; for pricing the intervention, the ANN's sharper probabilities might be worth their opacity. Calibration, as Chapter 18 insisted, is a business question wearing a statistician's coat.

5. The Business Metric

Now the chapter's climax, where everything becomes money. The agreed economics: a retention offer costs $200 per customer contacted; a saved customer is worth $2,000 of lifetime value. Assume (and state!) that an offer, when extended to a genuine churner, saves them; a contact to a stayer wastes $200. Against the do-nothing world — 405 test churners walk, $810,000 of exposure — the model's profit at threshold t is:

profit(t) = 2,000 · TP(t) − 200 · FP(t)

TP = churners called (saved), FP = stayers called (wasted offers). Every 0.01 of threshold trades some of one for some of the other; the optimum is where the trade stops paying.

The real sweep, on the champion's actual test probabilities:

Threshold t	0.04	0.08	0.09	0.10	0.20	0.50	0.90
Profit	$567.0k	$604.4k	$610.8k	$610.4k	$577.2k	$414.6k	$138.8k
Calls made	1521	1180	1115	1040	667	292	76
Churners caught	396	382	379	372	323	215	70

Figure 33.1 — The real profit curve. The empirical optimum (t = 0.09, $610,800) sits exactly where the cost ratio predicted — and the accuracy-default 0.5 threshold would have left 32% of the value on the table.

Three findings, in ascending order of importance. First: the optimum is t = 0.09 — call 1,115 of 2,000 customers, catch 379 of 405 churners (93.6%), net $610,800 against the $810,000 do-nothing exposure. Second: the default 0.5 threshold — the one "accuracy" implicitly endorses — nets $414,600: leaving $196,200, a third of the value, unclaimed. The single most expensive line in a naive pipeline is predict() where predict_proba() belonged. Third — the elegant one: the optimum was predictable without any sweep. The marginal customer is worth calling exactly when their churn probability clears offer-cost ÷ customer-value = 200/2000 = 0.10. The empirical 0.09 kisses the formula. The threshold belongs to the business; the model's job is to rank.

⚠

Warning — state the assumptions: the 100%-save assumption is generous (real offers save a fraction of recipients), and CLV is an average, not a constant. Re-run the sweep with "saves 40% of contacted churners" and the optimum shifts (t* = 0.10/0.4 = 0.25 — fewer, surer calls). The deliverable is the method plus a sensitivity note, not one brittle number.

6. What the Model Knows

The champion's gain-based importances — and how they rhyme with the dataset's own churn rates (Chapter 26's audit):

Feature (champion)	Importance	The dataset's own story
NumOfProducts	0.211	1 product: 27.7% churn · 2: 7.6% · 3: 82.7% · 4: 100%
Age	0.201	churners average 44.8 vs stayers 37.4
IsActiveMember	0.177	inactive 26.9% vs active 14.3%
Germany	0.104	32.4% churn vs France/Spain ≈ 16%
Gender · Balance · …	0.079 · 0.056 · …	secondary sharpeners

The model's attention follows the data's truth — the product-count cliff (nonlinear, tree-shaped: exactly why the ensembles beat logistic regression's straight line by nine AUC points), then age, then activity, then the Germany anomaly a linear model could never isolate from an interaction. And the deliverable falls out directly: the call list is the 1,115 customers above the 0.09 threshold — concentrated, by construction, among multi-product, older, inactive, German customers. The retention team now knows both who and, from the importances, likely why — the beginning of an intervention, not just a prediction.

🌎

Real World: this chapter's shape — shortlist, tune, test once, price the threshold, ship — is the weekly rhythm of applied ML teams. What varies by employer is only the currency: dollars of retention, cases of fraud, minutes of downtime, patients missed.

📋 Stable content — Reviewed: October 2026

7. Ship It — Deployment & the Workflow

The last professional habit: a model that lives only in a notebook has no value. The minimum deployable artifact is the fitted pipeline, saved with everything it needs to score a new customer:

# ---- the artifact: one object, versioned, self-contained
import joblib
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from xgboost import XGBClassifier

# preprocessing + model as ONE pipeline — no room for order mistakes
pre = ColumnTransformer(
    [('geo', OneHotEncoder(), ['Geography'])],
    remainder='passthrough')
pipe = Pipeline([
    ('pre', pre),
    ('scale', StandardScaler()),
    ('clf', XGBClassifier(n_estimators=300, max_depth=3,
                          learning_rate=0.05, subsample=0.8,
                          eval_metric='logloss')),
]).fit(df_train, y_train)          # raw columns in, probability out

joblib.dump({'pipeline': pipe, 'threshold': 0.09,
             'metadata': {'trained': '2026-10-04',
                          'cv_auc': 0.8638, 'test_auc': 0.8754,
                          'features': list(df_train.columns)}},
            'churn_champion_v1.joblib')

# ---- in production: probabilities, then the business threshold
p = pipe.predict_proba(new_customer)[:, 1]
call = p >= 0.09                   # not 0.5 — Section 5 decided this

Listing 33.1 — The deployment artifact: pipeline, threshold, and metadata in one versioned file.

And the course, compressed to the checklist a professional can run on any new problem — every line a chapter:

Figure 33.2 — The professional loop. Steps 1–7 shipped this chapter's model; step 8 writes the next chapter's brief.

💡

Pro Tip — the monitoring minimum: log every scored customer's features and probability; weekly, compare live score distributions and (once outcomes land) live precision/recall against the deployed metadata's numbers. When either drifts past an agreed margin — customer base changed, campaign changed — that is the next brief, and the loop restarts at step 1.

⚠ Volatile content — Reviewed: October 2026 · Next review: January 2027

8. The Road Ends — Epilogue

Where the industry around this workflow is heading — the volatile sign-off:

MLOps: experiment tracking (MLflow and kin), feature stores, model registries, CI/CD for models — the plumbing that turns Figure 33.2's loop from a checklist into an automated system.
Foundation models: pretrained transformers now start many projects (embeddings, text features, image perception) — and the tabular core this course taught still decides most business questions on top of them.
The constant: tools churn (this section exists to be reviewed quarterly); the discipline does not. CV inside train, test once, money sets the threshold, monitor everything — these outlive every framework.

The course, in ten sentences. Data becomes a model only through honest preprocessing (I). Numbers predict numbers under assumptions worth checking, and simple baselines win embarrassingly often (II). Classification is probability management, and the metric is a business decision (III). Structure without labels is real but must justify its clusters (IV–V). Feedback loops reward courage in exploration (VI). Language and images yield to representation learning (VII). Neural networks are calculus run backwards — and data beats architecture at small scale (VIII). Variance has axes, and labels can aim them (IX). Choosing and tuning models honestly is a discipline, not a vibe (X). And the whole of it fits in one loop — brief, audit, shortlist, tune, test once, threshold, deploy, monitor — which you have now walked end to end, on real data, to a real $610,800.

Where next, reader: the R edition of this course (every chapter's twin implementation); a personal project on data you care about, run through Figure 33.2's loop; the papers behind the chapters that grabbed you — Fisher 1936, Breiman 2001, Chen & Guestrin 2016, Vaswani 2017. The toolbox is yours. Build something worth measuring.

🛠 Hands-On: Run the Whole CapstoneCapstone

The retention team approved the project — conditionally. Reproduce the entire pipeline end to end, verify every headline number, then answer the two questions they will ask in the review: what happens to the threshold if offers only save 40% of contacted churners, and what would convince you to retrain the model next quarter?

Steps

Audit and prepare: drop identifiers, encode, split 8,000/2,000 (seed 0), scale on train only; verify the 20.25% test churn rate.
Shortlist: 10-fold CV on the four families with AUC + recall + accuracy; reproduce the Section 2 table.
Tune both finalists (the two grids); confirm 0.8638 vs 0.8620 and the winning parameter shape.
Evaluate once on test; reproduce cm [[1518, 77], [190, 215]] and the full metric suite.
Money: sweep thresholds; reproduce the $610,800 optimum at t = 0.09 and the $414,600 at 0.5; verify the t* = offer/CLV prediction.
Sensitivity: re-sweep with "offers save 40%" economics; report the new optimum and call count.
Ship: build the single-Pipeline artifact (raw columns in, probability out), joblib it with metadata, and write the monitoring plan's first weekly report template.

View Solution / Walkthrough

import numpy as np, pandas as pd, joblib
from sklearn.model_selection import (train_test_split, cross_validate,
                                     GridSearchCV)
from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.metrics import (confusion_matrix, accuracy_score,
                             precision_score, recall_score, f1_score,
                             roc_auc_score)

ds = pd.read_csv('Churn_Modelling.csv')
X = ds.iloc[:, 3:-1].values; y = ds.iloc[:, -1].values
le = LabelEncoder(); X[:, 2] = le.fit_transform(X[:, 2])
X = np.array(ColumnTransformer([('e', OneHotEncoder(), [1])],
         remainder='passthrough').fit_transform(X))
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.2, random_state=0)
sc = StandardScaler(); Xtr = sc.fit_transform(Xtr); Xte = sc.transform(Xte)

# ---- shortlist
for name, m in [('Logistic', LogisticRegression(max_iter=1000)),
                ('Tree', DecisionTreeClassifier(random_state=0)),
                ('Forest', RandomForestClassifier(random_state=0, n_jobs=-1)),
                ('XGB', XGBClassifier(eval_metric='logloss', n_jobs=-1))]:
    r = cross_validate(m, Xtr, ytr, cv=10, n_jobs=-1,
                       scoring=['roc_auc', 'recall', 'accuracy'])
    print('%-9s AUC %.4f±%.4f R %.4f acc %.4f'
          % (name, r['test_roc_auc'].mean(), r['test_roc_auc'].std(),
             r['test_recall'].mean(), r['test_accuracy'].mean()))

# ---- tune finalists (AUC, 5-fold)
gs_x = GridSearchCV(XGBClassifier(eval_metric='logloss', n_jobs=-1),
    {'n_estimators': [100, 300], 'max_depth': [3, 6],
     'learning_rate': [.05, .2], 'subsample': [.8, 1.0]},
    scoring='roc_auc', cv=5, n_jobs=-1).fit(Xtr, ytr)
gs_f = GridSearchCV(RandomForestClassifier(random_state=0, n_jobs=-1),
    {'n_estimators': [100, 300], 'max_depth': [None, 10],
     'max_features': ['sqrt', 0.5]},
    scoring='roc_auc', cv=5, n_jobs=-1).fit(Xtr, ytr)
print('XGB %.4f %s' % (gs_x.best_score_, gs_x.best_params_))   # .8638
print('RF  %.4f %s' % (gs_f.best_score_, gs_f.best_params_))   # .8620

# ---- test once
champ = gs_x.best_estimator_
p = champ.predict_proba(Xte)[:, 1]
yp = (p >= .5).astype(int)
print(confusion_matrix(yte, yp),
      'acc %.4f P %.4f R %.4f F1 %.4f AUC %.4f'
      % (accuracy_score(yte, yp), precision_score(yte, yp),
         recall_score(yte, yp), f1_score(yte, yp),
         roc_auc_score(yte, p)))

# ---- money sweep (+ 40%-save sensitivity)
def sweep(save_rate):
    def profit(t):
        q = (p >= t).astype(int)
        TP = int(((q==1)&(yte==1)).sum()); FP = int(((q==1)&(yte==0)).sum())
        return 2000*save_rate*TP - 200*FP, TP, FP
    ts = [t/100 for t in range(4, 95)]
    return max(((t,) + profit(t) for t in ts), key=lambda r: r[1])
print('100%% save:', sweep(1.0))   # t=0.09-ish, ~$610,800
print('40%%  save:', sweep(0.4))   # higher threshold, fewer calls

# ---- ship (pipeline form)
from sklearn.pipeline import Pipeline
pipe = Pipeline([('pre', ColumnTransformer(
                    [('e', OneHotEncoder(), [4])], remainder='passthrough')),
                 ('scale', StandardScaler()),
                 ('clf', champ)]).fit(ds.iloc[:, 3:-1], y)   # raw cols in
joblib.dump({'pipeline': pipe, 'threshold': 0.09},
            'churn_champion_v1.joblib')

Expected results (real): shortlist table as in Section 2 (Logistic AUC 0.763 · Tree 0.676 · Forest 0.851 · XGB 0.839); tuned 0.8638 (XGB, lr .05/depth 3/300/ subsample .8) vs 0.8620 (RF); test cm [[1518, 77], [190, 215]], acc 0.866, P 0.736, R 0.531, F1 0.617, AUC 0.875; profit sweep peaking $610,800 at t = 0.09 (1,115 calls, 379/405 churners) vs $414,600 at t = 0.5; with 40%-save economics the optimum moves to t ≈ 0.24–0.25 (≈ 200/(2000×0.4)) with roughly half the calls — the formula again. Monitoring answer sketch: drift in the scored-probability distribution, live recall below the deployed metadata's band, or any campaign/product change (the NumOfProducts cliff!) triggers the next brief.

🎯 Key Takeaways

ML work starts with a brief, not a model: question, data, metric, deliverable — and the imbalance/identifier traps are handled on day one.

Shortlist by CV, not vibes: AUC ranks the four families (forest 0.851, XGB 0.839 untuned), because the deliverable is a ranking and the threshold comes later.

Tune both finalists: tuned XGB 0.8638 edges tuned forest 0.8620 — a whisker inside the folds' spread; either would ship. Plateaus, again.

Test once: acc 0.866, recall 0.531, AUC 0.875 — edging Chapter 26's network on the same 2,000 customers, with the honest "by a handful" footnote.

The money sets the threshold: $200 offers, $2,000 customers → optimum t = 0.09, $610,800, 93.6% of churners caught. The formula t* = offer/CLV predicted it — and default 0.5 leaves a third behind.

The profession is a loop — brief, audit, shortlist, tune, test once, threshold, deploy, monitor — and the discipline outlives every tool this course taught.