Part III — Classification

Chapter 13 · Support Vector Machine (SVM)

Logistic regression accepts any separating line; K-NN draws no line at all. The Support Vector Machine asks a sharper question: among all lines that separate the classes, which one leaves the widest street between them — the most room before a new customer lands on the wrong side? Geometry with a purpose, priced by one dial (C), defined by a few support vectors, and — as this chapter's real run shows — utterly dependent on scaling.

📖 Chapter 13 of 33 ⏱ ~85 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Explain the maximum-margin principle and why the widest street generalises best
Identify support vectors by hand in a one-dimensional example, and connect them to Chapter 7's SVR machinery
Describe soft margins, slack variables, and C as the margin-versus-violations dial — with the real C study
Contrast hinge loss with logistic's log-loss, and explain SVM's "margin indifference"
Witness — in real accuracy, warnings and training time — why scaling is the SVM's oxygen
Read the real results and place the linear SVM among its classification rivals

1. The Widest Street

Chapter 11 drew a straight boundary wherever log-likelihood peaked; many lines separate the buyers from the non-buyers almost equally well. Chapter 12 let the neighbourhood decide locally. The Support Vector Machine (Cortes & Vapnik, 1995) reframes boundary choice as geometry: among all separating hyperplanes, choose the one whose margin — the empty street between the two classes — is widest.

Figure 13.1 — Many lines separate the classes (dashed); the SVM picks the one with the widest street (shaded), pinned by its support vectors.

Why insist on width? Because the street is the safety margin. A new customer arriving near the boundary must travel half the street's width before crossing to the wrong class — the widest street buys the most insurance against measurement noise and sampling luck. Margin width is robustness, made geometric.

📝

Note — the family reunion: this is Chapter 7's machine in classification clothes. There the tube ignored small errors in y; here the street ignores everything except the points nearest the boundary. Both models are decided by support vectors, both price violations with C, both speak kernels (Chapter 14). Learn one, own both.

2. Margins & Support Vectors

Strip the geometry to one dimension and every ingredient becomes checkable by hand. Two classes on a line: non-buyers at −3 and −2, buyers at +2 and +3 (think standardised salary):

Figure 13.2 — The 1D street: edges touch −2 and +2; the boundary sits at 0. The outer points could vanish without moving anything.

The reasoning, entirely by hand:

The street's edges must stop at the closest points: −2 and +2. Any wider and a class leaks in; any narrower wastes available room.
The boundary is the street's midpoint: 0. Street width: 4.
−3 and +3 are irrelevant. Delete them, refit: identical boundary. Only the closest points — the support vectors — matter.

In d dimensions the same logic yields the famous formula — the optimisation's twin faces:

margin width = 2 / ‖w‖

Maximising the margin is minimising ‖w‖ — the SVM's primal problem, a convex quadratic program. Smaller weights, wider street, smoother boundary: Chapter 6's complexity story retold as geometry. On our real data, the solution pins 128 of 300 training points as support vectors (64 per class) — 43%, because overlapping classes put many points on or inside the street (Section 3).

3. Soft Margins: the C Dial

Real classes overlap — some young high-earners buy, some older low-earners don't — so a hard margin (zero violations) does not exist on this data. The soft-margin SVM relaxes: points may enter the street or even cross the boundary, each paying a fine ξ (xi, the slack of Chapter 7's tube), with C setting the price per unit:

minimize ½‖w‖² + C · Σξ_i

C small — violations cheap: wide, placid street, many slackers, more bias. C large — violations expensive: the street narrows and contorts to obey the data, more variance. C is Chapter 6's dial wearing geometric clothes.

The real C study (same split and scaling, test and train accuracy with support-vector counts):

C	0.01	0.1	1 (source)	10	100
Test acc	0.87	0.89	0.90	0.89	0.89
Train acc	0.783	0.817	0.823	0.830	0.830
Support vectors	206	148	128	126	125

Read the last row as the street's personality meter: at C = 0.01 the tolerant street lets 206 points sit on or inside it; raise the fines and the sidewalk clears — 125 by C = 100. Accuracy moves gently (0.87 → 0.90 → 0.89): this dataset is near-linear, so the street barely needs to flex. The lesson is the shape of the response, not this particular dataset's plateau — noisy, non-separable data swings far more with C, which is why Chapter 31 tunes it systematically.

💡

Pro Tip — the SV census as diagnostic: n_support_ tells you how contested your boundary is. A handful of SVs: clean separation, robust solution. Half the dataset (our 128/300): the classes genuinely overlap, and expectations about achievable accuracy should adjust accordingly — a fact no accuracy number alone reveals.

4. Hinge Loss — the SVM Personality

Chapter 11 fitted logistic by log-loss; the SVM fits by hinge loss (labels coded ±1, f(x) the signed score):

L(y, f(x)) = max( 0, 1 − y · f(x) )

A point on the correct side beyond the street edge (y·f(x) ≥ 1) costs zero. Inside the street or on the wrong side, the cost grows linearly.

The contrast with log-loss is the SVM's personality in one comparison:

Log-loss never stops caring. However confidently right logistic is, a slightly larger margin would still shave the loss — it keeps polishing.
Hinge stops at the street edge. Once correctly classified with margin, the point is dead to the loss. Hence the sparsity: only street-dwellers and violators influence the fit.
Both are convex — reliable optimisation, no local traps (Chapter 11's superpower, shared).

⚠

Warning — no free probabilities: hinge optimises the boundary, not belief. SVC by default offers no predict_proba — only decision_function: the signed distance to the boundary (real values, e.g. −1.57, +0.21 in Section 7). Distances are confidence-like but not probabilities; producing honest probabilities requires extra calibration — a Chapter 18 topic. When the business needs "72% likely to buy", logistic remains the natural tongue.

5. Scaling: the Loudest Lesson Yet

Chapter 3's table has marked SVMs scale-mandatory since day one; Chapter 7's SVR measured the damage; this chapter's run is the most dramatic of the course — because the street itself is measured in feature units. A street built from ages and raw rupees allocates essentially all its width to the rupee axis; and the optimiser, grinding over numbers tens of thousands strong, crawls.

The real head-to-head, identical code otherwise:

	Scaled	Unscaled (raw ₹ units)
Test accuracy	0.90	0.33 — worse than always-guess-"no" (0.64)
Training time	instant (< 0.1 s)	minutes — our unbounded attempt had to be killed; the capped run quit at 200 iterations
sklearn's verdict	—	ConvergenceWarning: "Consider pre-processing your data with StandardScaler or MinMaxScaler"
Support vectors	128	10 (a degenerate solution)

Three failures in one: an accuracy below the majority-class floor, a solver that melts down, and a warning that names the cure. For K-NN the unscaled model quietly lost ten points (Chapter 12); for the SVM it collapses outright and announces it. Same root cause — geometry needs comparable units — with the loudest possible symptoms.

🌎

Real World: "My SVM has been training for an hour" is one of the most common beginner cries on practitioner forums, and the canonical reply is a one-liner: did you scale? Now you can answer it with a mechanism, not folklore — and note that the same caution applies whenever SVMs meet rupees, milliseconds, or pixel counts.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.svm import SVC
classifier = SVC(kernel='linear', random_state=0)
classifier.fit(X_train, y_train)

print(classifier.predict(sc.transform([[30, 87000]])))    # [0]

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)                                 # [[66  2]
                                          #  [ 8 24]]
print(accuracy_score(y_test, y_pred))     # 0.9

# The SVM's own report card
print(classifier.n_support_)              # [64 64] — support vectors per class
print(classifier.support_.size)           # 128 of 300
print(classifier.decision_function(
    sc.transform([[30, 150000]])))        # [0.21] — just inside the buyer side

Listing 13.1 — The complete linear-SVM pipeline (source: support_vector_machine.py).

# The C-dial study of Section 3
for C in [0.01, 0.1, 1, 10, 100]:
    m = SVC(kernel='linear', C=C).fit(X_train, y_train)
    print(f'C={C:<6g} test={m.score(X_test, y_test):.4f} '
          f'SVs={m.support_.size}')

# The cautionary tale — bounded, so it cannot ruin your afternoon
from sklearn.exceptions import ConvergenceWarning
raw = SVC(kernel='linear', C=1, max_iter=200).fit(X_raw_train, y_train)
# -> ConvergenceWarning + accuracy 0.33

Listing 13.2 — The C study and the (safely capped) unscaled catastrophe.

💡

Pro Tip: For large datasets, SVC(kernel='linear') scales poorly (roughly n²–n³); reach for LinearSVC or SGDClassifier(loss='hinge') — same hinge, industrial size. Kernel SVMs (next chapter) stay on small-to-medium data.

7. Three Models, One Customer

The real scoreboard, same data, same split, same scaling:

	Logistic (Ch 11)	SVM linear (this chapter)	K-NN (Ch 12)
Confusion matrix	[[65, 3], [8, 24]]	[[66, 2], [8, 24]]	[[64, 4], [3, 29]]
Accuracy	0.89	0.90	0.93
Errors	11	10 (2 FP + 8 FN)	7
Boundary	Straight (log-loss)	Straight (max margin)	Local, wiggly
Confidence output	Calibrated probability	Signed distance	Vote share

The linear SVM edges logistic (0.90 vs 0.89) with almost the same boundary family — a reminder that different losses on the same geometry give sibling, not twin, models (hinge's indifference shifts the line slightly; note its 2 false positives to logistic's 3). Both are beaten by K-NN's flexible vote, confirming the running diagnosis: this data's class region is not linearly separable.

7.1 The boundary customer, third opinion

Customer (30, ₹150k) — the young high-earner — has now been examined by three philosophies, all real: logistic sat on its fence (p = 0.491); K-NN's neighbourhood voted BUY; the SVM places the customer at decision value +0.21 — inside the buyer side, but barely past the street's edge. Three shades of the same verdict, and a fourth model would add another. When a decision this fine matters, the professional move is Chapter 31's — gather evidence across models and tunings systematically, not by collecting opinions one at a time.

💡

Pro Tip — reading w: the fitted weights w = [1.60, 0.97] (scaled) say age pushes the signed distance about 1.6× harder than salary per SD — consistent with Chapter 11's odds ratios (7.98 vs 3.03). Two loss functions, one dataset, same underlying story: different microscopes, same specimen.

8. Strengths, Limits & the Road Ahead

Strengths	Limitations
Max-margin robustness — geometry as insurance	Linear street only, in this chapter — curved regions need kernels
Convex optimisation — one global optimum, reliably found	Scaling mandatory: 0.33 accuracy and solver meltdown when ignored
Sparsity: support vectors alone define the model	No native probabilities — distances, not beliefs
Excellent in high-dimensional feature spaces	Training cost n²–n³ — big data needs LinearSVC/SGD variants

The scoreboard's message is structural: K-NN's local flexibility beat the linear street, 93 to 90. The SVM's answer to curved boundaries is not to abandon the street — it is to move the fight to a space where the street works. Chapter 14's kernel trick lifts the data into higher dimensions (implicitly, at no column cost — Chapter 6's promise, fulfilled) where a linear separator carves curved regions back home. Same support vectors, same C, transformed geometry — the strongest idea in this Part of the course.

🛠 Hands-On: Survey the StreetBeginner

Fit the linear SVM, census its support vectors, run the C study, stage the scaling catastrophe (safely), and file the three-model report on the boundary customer.

Steps

Run the Section 6 pipeline; verify [[66, 2], [8, 24]] and 0.90.
Report n_support_ per class and the SV share of training data. What does 43% tell you about the classes' overlap?
Run the C study {0.01, 0.1, 1, 10, 100}; plot (in a table) accuracy vs SV count. Where does the street clear?
Run the unscaled SVM with max_iter=200; record accuracy, the warning, and the SV count. Explain all three symptoms via units.
Collect decision_function values for (30, ₹87k), (30, ₹150k), (45, ₹87k); interpret each sign and magnitude.
Write the three-model verdict on (30, ₹150k): logistic 0.491, SVM +0.21, K-NN "buy" — which commitment would you act on, and why?

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, accuracy_score

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
Xs_tr, Xs_te = sc.fit_transform(X_tr), sc.transform(X_te)

clf = SVC(kernel='linear', random_state=0).fit(Xs_tr, y_tr)
print(confusion_matrix(y_te, clf.predict(Xs_te)), clf.score(Xs_te, y_te))
print('n_support_', clf.n_support_, ' total', clf.support_.size)

for C in [0.01, 0.1, 1, 10, 100]:
    m = SVC(kernel='linear', C=C).fit(Xs_tr, y_tr)
    print('C=%-6g test=%.3f SVs=%d' % (C, m.score(Xs_te, y_te), m.support_.size))

raw = SVC(kernel='linear', max_iter=200).fit(X_tr, y_tr)
print('unscaled:', raw.score(X_te, y_te), raw.support_.size)

for cust in [[30, 87000], [30, 150000], [45, 87000]]:
    print(cust, clf.decision_function(sc.transform([cust]))[0].round(2))

Expected results (real):

Step 2: [64, 64] — 128/300 = 43%: near half the training data lives on the street; genuine class overlap, so accuracy near 0.90 is close to this boundary family's ceiling.
Step 3: C 0.01→1: SVs 206→128, test 0.87→0.90; beyond C=1 the census barely clears (126, 125) and accuracy drifts to 0.89 — the tolerant-to-strict sweep in one table.
Step 4: unscaled: 0.33 accuracy, 10 SVs, ConvergenceWarning — the street collapsed onto the salary axis and the solver surrendered; all three symptoms share one cause (units).
Step 5: −1.57 (comfortably no), +0.21 (barely yes — the street's edge), +0.81 (solidly yes).
Step 6: a defensible answer: act on K-NN's buy (the most accurate model) but flag the customer as borderline; the disagreement itself is information — the formal treatment is Chapter 18's thresholds and Chapter 31's ensembles of evidence.

🎯 Key Takeaways

Among all separating lines, the SVM picks the widest street — margin as insurance: the most room before a new point crosses wrong.

Support vectors alone define the boundary. In 1D: street (−2, +2), boundary 0, outer points irrelevant — and 128 of 300 points on the real street.

C prices street violations: small C = wide tolerant street (206 SVs), large C = narrow obedient one (125). A bias-variance dial in geometric dress.

Hinge loss stops caring at the street edge — sparsity by design; log-loss polishes forever. Distances, not probabilities, are the SVM's native tongue.

The scaling catastrophe: 0.90 → 0.33, solver meltdown, official warning. The street is measured in feature units — comparable units or bust.

0.90 sits between logistic's 0.89 and K-NN's 0.93: the region isn't linearly separable — the cue for Chapter 14 to bend space itself.