Part III — Classification

Chapter 11 · Classification Foundations & Logistic Regression

Part III asks a new question: not how much, but which one. Will this customer buy — yes or no? We open with logistic regression: regression's skeleton wearing a sigmoid, the most interpretable classifier ever built. Odds and log-odds made concrete by hand, coefficients read as odds ratios, the assumption suite (shorter than OLS's, differently shaped), and the threshold that turns probabilities into decisions — a choice, not a law.

📖 Chapter 11 of 33 ⏱ ~95 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Define classification and demonstrate — with real numbers — why linear regression cannot do it
Compute sigmoid values by hand and convert between probability, odds and log-odds
Read logistic coefficients as odds ratios, including the per-standard-deviation caveat on scaled features
Explain maximum likelihood estimation and its companion loss, log-loss (binary cross-entropy)
State the assumption suite of logistic regression — and which OLS assumptions it drops
Interpret the confusion matrix and the decision threshold, with real numbers from Social_Network_Ads.csv

1. From "How Much" to "Which One"

Classification is supervised learning with a categorical target: spam or not-spam, churn or stay, purchase or scroll past. Part III's running dataset is Social_Network_Ads.csv — 400 social-network users, their Age (18–60) and EstimatedSalary (₹15k–₹150k), and whether they Purchased the advertised SUV:

Quantity	Value	Reading
Rows	400	300 train / 100 test (source split: 0.25, seed 0)
Features	Age, EstimatedSalary	Both numeric — standardise (Section 7)
Target	Purchased (0/1)	Binary classification
Class balance	143 buyers = 36%	Moderately imbalanced — remember this number for Section 8's accuracy caveat

1.1 Why linear regression cannot classify

The tempting hack: encode the classes as 0 and 1, fit Chapter 4's OLS, round the predictions. We ran it for real. The fitted line predicts values from −0.31 to +1.19 — outside [0, 1] on both sides. What could "−0.31 probability of buying" possibly mean?

Three structural failures, all visible in that one number:

No output bounds. A line extends forever; probabilities live in [0, 1].
Wrong noise model. A 0/1 target's variance depends on its mean (p(1−p)) — Chapter 4's equal-variance assumption is violated by construction.
Balance sensitivity. Add many non-buyers and the line tilts, changing predictions for everyone — a probability model should be stable under class re-weighting.

The fix is not a new mindset — it is a transformation. Keep the linear score b₀ + b₁x₁ + b₂x₂, but pipe it through a function that squashes the whole real line into (0, 1). Enter the sigmoid.

2. The Sigmoid: Scores into Probabilities

σ(z) = 1 / (1 + e^−z)

Any real z maps to (0, 1): symmetric about (0, ½), asymptotic to 0 and 1 — never touching either. Steepest at z = 0, where a unit of score buys the most probability.

Five values by hand — internalise these anchors and you can eyeball any logistic output:

z (score)	−4	−2	0	+2	+4
σ(z)	0.018	0.119	0.500	0.881	0.982

Figure 11.1 — The sigmoid: linear score in, probability out; the decision coin flips at z = 0.

The full model — note that it stays linear in a precise sense (Section 3):

P(yes | x) = σ( b₀ + b₁x₁ + b₂x₂ )

3. Odds, Log-Odds & the Coefficient Reading

Probabilities are bounded; the linear score is not. The bridge is the logit — and its cousin odds, the bookmaker's currency:

odds = p / (1 − p) logit(p) = ln(odds) ⟹ logit(p) = b₀ + b₁x₁ + b₂x₂

Logistic regression is linear regression of the log-odds. Log-odds run from −∞ to +∞ — exactly the range a linear score can fill.

The conversion table, every entry by hand:

p	0.10	0.25	0.50	0.75	0.881	0.90
odds	1/9	1/3	1	3	7.4	9
logit	−2.20	−1.10	0	+1.10	+2	+2.20

3.1 The odds-ratio reading — the payoff

Exponentiate a coefficient and you get an odds ratio — the multiplicative effect on the odds of one unit of that feature. The real fitted model (Section 7, scaled features):

b_age = 2.077 ⇒ e^2.077 = 7.98 · b_salary = 1.110 ⇒ e^1.110 = 3.03

Age (scaled): one standard deviation more age (≈ 10.7 years) multiplies the odds of purchase by 7.98. Age is the dominant driver.
Salary (scaled): one SD more salary (≈ 40k) multiplies odds by 3.03.
Beware the units: these are per-standard-deviation effects because the features were scaled — not per year or per rupee (Section 7's caveat). On raw features the coefficients would be per-unit but the fit would differ numerically.

💡

Pro Tip — odds ≠ probability: "odds multiply by 8" does not mean "probability ×8". From p = 0.11 (odds 1/8), multiplying odds by 8 gives odds 1 → p = 0.5 — probability merely quintupled. Always convert back before speaking to stakeholders: odds ratios are for effects, probabilities for decisions.

4. How Coefficients Are Learned: MLE

OLS had a closed form — the normal equations. Logistic regression does not: the sigmoid's curvature denies algebra a tidy solution, so coefficients are found by maximum likelihood estimation (Chapter 2's vocabulary, now doing real work): choose the b's that make the observed labels most probable.

What that means concretely, for one customer at a time:

The model says p = 0.9; the customer bought (y = 1) → likelihood contribution 0.9 — good.
The same p = 0.9; the customer didn't (y = 0) → contribution 0.1 — the model is punished by exactly (1 − p).
Total likelihood = product over all 300 training rows; maximise it (equivalently, minimise its negative log).

LogLoss = −(1/n) Σ [ y_i·ln(p_i) + (1−y_i)·ln(1−p_i) ]

This is binary cross-entropy — the same loss the neural networks of Chapter 26 will minimise, and the same "distance" ROC curves integrate over in Chapter 18. sklearn solves it with efficient iterative solvers (no random_state drama here — the problem is convex: one optimum, found reliably).

📝

Note: Convexity is logistic regression's quiet superpower: unlike polynomial degree, tree depth or SVR's C-ε-γ triple, the core fit has no local optima to get lost in. Its main dial — regularization strength C — arrives quietly with sklearn's default (L2, mild). Chapter 31 tunes it when it matters.

5. The Assumption Suite

Laerd-style, the conditions behind trustworthy logistic coefficients — notice how the suite differs from OLS's LINE:

Assumption	What it demands	How to check	If violated
Correct outcome type	Binary (or multinomial via extensions) target encoded 0/1	By construction here	Wrong model family entirely
Independence of observations	Rows don't carry shared information (same user twice, clustered samples)	Data-collection audit; duplicates check	Overconfident standard errors
No severe multicollinearity	Features not near-linearly related	Chapter 5's VIF (thresholds 5/10)	Unstable coefficients, misleading odds ratios
Linearity of log-odds	Each numeric feature relates linearly to the logit — not to the probability	Box–Tidwell test; binned empirical logit plots	Biased coefficients; consider transformations or splines
Adequate sample size	Rule of thumb: ~10 events per predictor (minority class!)	min(n₁, n₀)/p ratio	Separation: perfect-but-absurd coefficients

⚠

Warning — what is NOT required: logistic regression needs neither normality of errors nor homoscedasticity — the two assumptions OLS lives or dies by. Interviewers love this distinction: OLS's residuals are Gaussian actors; logistic's 0/1 outcomes are Bernoulli by nature. What replaces them: the log-odds linearity above, and enough events to estimate effects. (400 rows with 143 buyers ÷ 2 predictors ≈ 71 events per predictor — comfortable.)

6. The Decision Boundary & the Threshold

A probability is not yet a decision. The rule that converts it: predict "yes" when p ≥ threshold, with the default threshold 0.5. Geometrically, p = 0.5 is exactly where the linear score z = 0 — so the decision boundary is the straight line b₀ + b₁x₁ + b₂x₂ = 0 in feature space:

Figure 11.2 — The fitted boundary on real anchors: young/low-salary → no; older/high-salary → yes; (30, ₹150k) sits at p = 0.49, exactly on the line.

And because the threshold is a choice, we ran the real experiment — same fitted model, 100 test rows, three thresholds:

Threshold	Accuracy	Predicted "yes" count	Personality
0.3	0.87	39	Aggressive — casts a wide net for buyers
0.5 (default)	0.89	27	Balanced
0.7	0.88	22	Conservative — only near-certain buyers

Same model, three different marketing campaigns. If a missed buyer costs a ₹50,000 margin and a wasted ad costs ₹50, the 0.3 threshold probably wins — a calculation Chapter 18 will formalise with precision and recall. For now, the point stands: 0.5 is a default, not a law.

📋 Stable content — Reviewed: August 2026

7. Implementation in Python

The source pipeline — split, scale (yes, even for logistic: the regularised solver and coefficient comparability both want it), fit, evaluate:

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values        # Age, EstimatedSalary
y = dataset.iloc[:, -1].values         # Purchased 0/1

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)          # 300 / 100

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)   # fit on train only (Ch 3)
X_test = sc.transform(X_test)

from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state=0)
classifier.fit(X_train, y_train)

# A single prediction — with its probabilities
print(classifier.predict(sc.transform([[30, 87000]])))          # [0]
print(classifier.predict_proba(sc.transform([[30, 87000]])))    # [[0.887, 0.113]]

# The confusion matrix — the classification scoreboard
from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)                                   # [[65  3]
                                            #  [ 8 24]]
print(accuracy_score(y_test, y_pred))       # 0.89

Listing 11.1 — The complete pipeline (source: logistic_regression.py).

# Coefficients as odds ratios (scaled features!)
import numpy as np
print('b0 =', classifier.intercept_[0].round(3))
print('b1, b2 =', classifier.coef_[0].round(3))       # [2.077, 1.110]
print('odds ratios =', np.exp(classifier.coef_[0]).round(2))  # [7.98, 3.03]

# The threshold experiment from Section 6
probs = classifier.predict_proba(X_test)[:, 1]
for t in [0.3, 0.5, 0.7]:
    print(t, ((probs >= t).astype(int) == y_test).mean())

Listing 11.2 — Reading the fit: odds ratios and threshold sweep.

💡

Pro Tip: The source also draws the boundary with a contourf prediction map — a fine habit. The modern equivalents, DecisionBoundaryDisplay.from_estimator(...) and ConfusionMatrixDisplay(...), do it in one call each; Chapter 18 adopts them permanently.

8. Reading the Results & the Road Ahead

8.1 The confusion matrix, cell by cell

	Predicted
Actual	No	Yes
No (65+3)	65 True Negative — correctly ignored	3 False Positive — ad wasted
Yes (8+24)	8 False Negative — buyer missed	24 True Positive — sale captured

89 of 100 test rows correct → accuracy 0.89. The errors split asymmetrically: 8 missed buyers versus 3 wasted ads. Whether that trade is right depends on the costs of each — the question accuracy silently ignores.

⚠

Warning — the accuracy floor: 64% of users did not buy; a model that predicts "no" for everyone scores 64% while being useless. Our 89% clears that floor comfortably, but as imbalance grows the floor rises and accuracy misleads — Chapter 18's precision, recall and ROC exist for exactly this reason.

8.2 Where logistic sits

Strengths: calibrated probabilities, odds-ratio interpretability, convex training, speed. Limits: a straight boundary — real buyer regions are rarely linearly separable, and the next six chapters progressively bend, localise and ensemble that boundary. First stop: Chapter 12's K-Nearest Neighbors, which abandons equations entirely and lets the nearest neighbours vote — the scale sensitivity Chapter 3 promised finally becomes vivid.

🛠 Hands-On: Probabilities into DecisionsBeginner

Marketing wants three customer scores — (30, ₹87k), (45, ₹87k), (30, ₹150k) — plus a recommendation on which threshold to campaign with. Fit the model, extract the probabilities, run the threshold study, and deliver a one-page recommendation.

Steps

Run the Section 7 pipeline; verify the confusion matrix [[65, 3], [8, 24]] and accuracy 0.89.
Extract predict_proba for the three customers. Which sits on the boundary, and why (age vs salary pulls)?
Convert the three probabilities to odds and log-odds by hand (Section 3's table as reference).
Report the odds ratios; explain to a non-technical colleague what "odds ×8 per SD of age" means — and what it does not.
Run thresholds {0.3, 0.5, 0.7}; tabulate accuracy and predicted-yes counts.
Recommend a threshold under two cost scenarios: (a) ads nearly free, (b) ads expensive. Justify with the counts.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
clf = LogisticRegression(random_state=0).fit(sc.fit_transform(X_tr), y_tr)

for cust in [[30, 87000], [45, 87000], [30, 150000]]:
    p = clf.predict_proba(sc.transform([cust]))[0, 1]
    print(cust, 'p=%.3f  odds=%.2f  logit=%.2f' % (p, p/(1-p), np.log(p/(1-p))))

probs = clf.predict_proba(sc.transform(X_te))[:, 1]
for t in [0.3, 0.5, 0.7]:
    yp = (probs >= t).astype(int)
    print('t=%.1f acc=%.3f yes=%d' % (t, (yp == y_te).mean(), yp.sum()))

Expected results (real):

Step 2: (30, 87k) p = 0.113 → no; (45, 87k) p = 0.735 → yes; (30, 150k) p = 0.491 — the boundary case: young (age pulls no) but wealthy (salary pulls yes), the forces nearly cancel.
Step 3: odds ≈ 0.13 / 2.78 / 0.96; logits ≈ −2.06 / +1.02 / −0.04 (the last ≈ 0, confirming "on the boundary").
Step 4: age e^2.077 = 7.98× odds per SD; salary 3.03× per SD — "not" including: it is not a probability multiplier and it is per SD, not per year.
Step 5: 0.3 → 0.87 acc, 39 yes; 0.5 → 0.89, 27; 0.7 → 0.88, 22.
Step 6: (a) ads cheap → 0.3 (capture more of the 32 actual buyers; accept 39 contacts); (b) ads expensive → 0.7 (22 contacts, highest precision). The full cost calculus is Chapter 18's precision/recall machinery — this is its seed.

🎯 Key Takeaways

Classification predicts categories; OLS on 0/1 labels fails structurally — real fit ranged −0.31 to 1.19, outside probability's home.

Logistic regression = linear regression of the log-odds: σ(z) = 1/(1+e⁻ᶻ) squashes the score; anchors σ(0)=0.5, σ(2)=0.881.

eᵇ is an odds ratio: our fit multiplies purchase odds by 7.98 per SD of age. Odds are for effects; convert to probability for decisions.

MLE fits by maximising label likelihood — minimising log-loss (binary cross-entropy), convexly. No normality or homoscedasticity required.

The boundary where p = 0.5 is a straight line in feature space; the threshold is a business choice — real sweep: 0.3→39 yes, 0.5→27, 0.7→22.

The confusion matrix is the scoreboard: [[65, 3], [8, 24]] → 89% — but 8 missed buyers vs 3 wasted ads is a cost question accuracy can't answer (Ch 18 will).