Part III — Classification

Chapter 14 · Kernel SVM

The linear street topped out at 0.90 because buyers and non-buyers form curved regions. The naive cure — engineer curved features by hand — explodes computationally. The kernel trick is the escape: lift the data into a vast (even infinite-dimensional) space where a straight street separates it, and pay for none of the columns. One idea, worked by hand, then proven in real numbers — where the curved street ties K-NN's accuracy with a fraction of the machinery.

📖 Chapter 14 of 33 ⏱ ~90 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Show, by hand, how lifting features to a higher dimension can make inseparable classes separable
Explain the kernel trick: why the SVM needs only inner products, and how kernels supply them without building the lifted features
Navigate the kernel menu — linear, polynomial, RBF, sigmoid — and read the real comparison on this dataset
Use γ (gamma) as the locality dial, recognising the underfit–sweet–overfit arc in the real sweep
Connect the curved street to the linear one: support vectors, C, scaling — what carries over, what changes
Choose kernels defensibly and know when to hand the decision to systematic tuning (Chapter 31)

1. The Problem: Streets That Cannot Be Drawn

Chapter 13 ended on an honest scoreboard: K-NN's flexible vote (0.93) beat the linear SVM's straight street (0.90) because the purchase region simply is not linearly separable — young high-earners and older low-earners bend both frontiers. No arrangement of one straight line, however wide its margin, carves the true region. The street must curve.

The naive fix is already in your toolkit: Chapter 6 manufactured curved features (x², x³, x⁴) and let linear regression bend. Why not the same for the SVM — build lifted columns [x₁, x₂, x₁², x₂², x₁x₂, …] and draw a straight street there? The idea is correct, and it dies on arithmetic: useful lift maps explode in columns (all monomials of degree 10 in 100 features ≈ 42 billion columns), and the most powerful lifts are infinite-dimensional — unbuildable by definition. Before the chapter's escape, watch the naive idea actually work, in miniature, by hand. The trick deserves it.

2. The Lifting Trick, by Hand

A one-dimensional dataset with two classes — an inner class B sitting at x = 0, and an outer class A on both wings at x = −2 and x = +2:

Figure 14.1 — The lifting trick in miniature: squaring x folds both wings of A onto one side, leaving B alone on the other.

The whole computation, every step by hand:

Point x	Class	Lifted z = x²
−2	A (outer)	4
0	B (inner)	0
+2	A (outer)	4

In x: any threshold (say "x > 1") labels the right wing A but the left wing A wrong; "inside vs outside" is not a 1-D half-space. Inseparable.
In z: A lands at 4 (twice), B at 0. The threshold z = 2 separates perfectly — and in the lifted (z) space that threshold is a linear street.
Project back down and the "straight" threshold z = 2 becomes the pair of cuts x = ±√2 — a curved region in the original space, built from a linear separator upstairs.

📝

Note — the general law: lift to enough dimensions and any reasonable labelling becomes separable by a hyperplane (the mathematics behind this is representer theory's neighbourhood). The catch is cost: building, storing and inner-producting billions (or ∞) of columns. Which is precisely the cost the next section deletes.

3. The Kernel Trick: Lifting Without Lifting

Look closely at what the SVM actually computes. Training needs the lifted data only through pairwise inner products ⟨φ(a), φ(b)⟩ — how aligned two lifted points are. Individual lifted columns? Never touched. The street is built entirely from point-to-point comparisons.

That observation is the crack in the wall:

K(a, b) := ⟨φ(a), φ(b)⟩

If you can compute the inner product directly from the original points — a closed-form shortcut that evaluates to the same number as the lifted inner product — you never need to build φ at all. That shortcut is a kernel. The street is fitted in the vast lifted space, at the cost of an n×n table of K-values computed in the original dimensions.

The classic proof it works, with our own example: the polynomial kernel K(a, b) = (a·b + 1)² on one-dimensional inputs multiplies out to 1 + 2ab + a²b² — exactly the inner product of the lifted vectors φ(x) = (1, √2·x, x²). Same number, zero columns built. And the star of this chapter goes further:

RBF: K(a, b) = exp( −γ · ‖a − b‖² )

Chapter 7's formula, now fully understood: the RBF kernel is the inner product of two points lifted into an infinite-dimensional feature space — a lift that could never be built column by column, yet whose pairwise comparisons cost one line of code. This is Chapter 6's promise ("flexibility through features") taken to its logical extreme: all the features, none of the storage.

💡

Pro Tip — what "trick" actually means: nothing sneaky. The boundary found upstairs is a perfectly ordinary linear street; projected back to the original space it carves curved regions (as Figure 14.1's threshold became two cuts). All the Chapter 13 machinery — support vectors, C, convex optimisation — carries over unchanged; only the geometry it acts in has quietly become infinite-dimensional.

4. The Kernel Menu

sklearn's kernel= parameter selects the lift. The menu, with the real scoreboard on this dataset (same split and scaling, C = 1 throughout):

Kernel	What it lifts to	Extra dials	Test acc (real)	Train acc	SVs
`linear`	No lift — the street at home (Ch 13)	—	0.90	0.823	128
`poly` (deg 3)	All monomials up to degree d	degree, coef0, γ	0.86	0.833	125
`poly` deg 2	Squares and cross-terms	degree=2	0.74	0.813	154
`rbf`	Infinite-dimensional bumps	γ	0.93	0.913	88
`sigmoid`	Neural-net-flavoured map	coef0, γ	0.86	0.833	125

Three real readings from that table:

RBF wins decisively — 0.93, tying K-NN's flexibility with a trained, compact boundary (88 support vectors; more on the leanness in Section 7).
Kernel choice is a modelling decision, not a formality. The degree-2 polynomial — a natural first guess for "curved" — collapsed to 0.74, worse than no lift at all. Its global parabola shape fights this data's local bumps; RBF's local bumps fit them.
Sigmoid (an historical import from neural networks) matches poly at 0.86 here — respectable, rarely the winner; keep it in the menu, not in the default slot.

⚠

Warning — small dataset, honest caveat: these gaps are measured on 100 test rows; a 3–4 point difference (0.86 vs 0.90) is within noise. The durable conclusions are structural (RBF's flexibility, poly-d2's mismatch), and the ranking should be confirmed by cross-validation before any deployment decision — exactly Chapter 31's job.

5. γ: the Locality Dial

RBF's similarity exp(−γ‖a−b‖²) decays with distance — and γ sets the decay rate: how far a training point's influence reaches. Chapter 7 previewed this for regression; here is the classifier's real sweep (C = 1, one γ at a time):

γ	0.01	0.1	0.5 (= 'scale')	1	10	100
Test acc	0.89	0.92	0.93	0.93	0.93	0.82
Train acc	0.793	0.893	0.913	0.913	0.937	0.973
Support vectors	186	115	88	88	176	271

Read it as three regimes — Chapter 6's bias-variance arc in one dial:

γ = 0.01 (too global): every point influences almost every other — the street flattens toward a near-linear boundary (0.89, close to the linear SVM's own 0.90), underfitting with 186 SVs of shared, muddy influence.
γ = 0.5–1 (the plateau): bumps are local enough to trace the curved region but wide enough to average noise — test peaks at 0.93 with the leanest census (88 SVs). Note gamma='scale' computes 1/(n_features × Var) = 0.5 here: the default lands on the plateau by design.
γ = 100 (too local): each point becomes its own island of similarity. Training climbs to 0.973 while test crashes to 0.82 — each new customer resembles almost nothing in memory, so the vote degenerates. The census explodes to 271: nearly everyone is a support vector when no one agrees with anyone.

💡

Pro Tip — the two-dial problem: RBF SVMs really tune two dials together (γ for locality, C for violation pricing from Chapter 13) — their interaction is real (high C can rescue a slightly-too- local γ, and vice versa). Manual one-at-a-time sweeps teach the physics; deployment tuning belongs to the systematic C × γ grid of Chapter 31.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

One character changes from Chapter 13's listing — and with it, the geometry:

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)      # STILL MANDATORY — kernels compute distances
X_test = sc.transform(X_test)

from sklearn.svm import SVC
classifier = SVC(kernel='rbf', random_state=0)
classifier.fit(X_train, y_train)

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)                                 # [[64  4]
                                          #  [ 3 29]]
print(accuracy_score(y_test, y_pred))     # 0.93
print(classifier.n_support_)              # [44 44] — 88 support vectors

# Confident distances for the anchor customers
print(classifier.decision_function(
    sc.transform([[30, 150000]])))        # [1.13] — firmly buyer-side

Listing 14.1 — The kernel switch: SVC(kernel='rbf') (source: kernel_svm.py).

# The menu and the dial, as code
for k in ['linear', 'poly', 'rbf', 'sigmoid']:
    m = SVC(kernel=k).fit(X_train, y_train)
    print(f'{k:8s} test={m.score(X_test, y_test):.3f}  SVs={m.support_.size}')

for g in [0.01, 0.1, 0.5, 1, 10, 100]:
    m = SVC(kernel='rbf', gamma=g).fit(X_train, y_train)
    print(f'gamma={g:<5g} test={m.score(X_test, y_test):.3f}  '
          f'train={m.score(X_train, y_train):.3f}  SVs={m.support_.size}')

Listing 14.2 — Reproduce Sections 4 and 5: the kernel menu and the γ sweep.

💡

Pro Tip: gamma='scale' (the default since sklearn 0.22) sets γ = 1/(n_features × Var(X)) — data-adaptive, and it landed on the plateau (0.5) here by construction. The older default 'auto' (γ = 1/n_features, ignoring variance) is a historical footgun you may still meet in old code.

7. Reading the Results: the Curved Street

The RBF street is still a street — support vectors, margins, hinge loss, all of Chapter 13 — but projected into age–salary space it bends around the buyers' region instead of slicing it. The real four-model scoreboard:

	Logistic (11)	SVM linear (13)	K-NN (12)	SVM RBF (14)
Confusion matrix	[[65, 3], [8, 24]]	[[66, 2], [8, 24]]	[[64, 4], [3, 29]]	[[64, 4], [3, 29]]
Accuracy	0.89	0.90	0.93	0.93
Boundary	Straight	Straight	No boundary (local votes)	Curved, trained
Model size	3 coefficients	128 SVs	All 300 rows	88 SVs
Missing only	—	8 buyers	—	3 buyers + 4 non-buyers

Two structural readings. First, the curved street matches K-NN's accuracy with a third of the memory — 88 stored support vectors versus K-NN's full 300-row lookup, and unlike K-NN's per-prediction search, this model is a finished object: the infinite-dimensional lifting happened once, at training, and survives compressed in those 88 points. Second, note which errors remain: 3 missed buyers and 4 courted non-buyers — the curved street reshapes the error profile, not just the count, and Chapter 18 will read exactly this trade with precision and recall.

7.1 The boundary customer, fourth opinion

Customer (30, ₹150k) again — the young high-earner every model has now met: logistic hedged (p = 0.491), the linear SVM tiptoed over the line (+0.21), K-NN's neighbourhood voted buy, and the RBF street commits firmly: decision value +1.13 — comfortably buyer-side, because the curved street wraps the wealthy-young cluster that a straight line had to bisect. Four models, one dataset, an increasingly confident consensus — and the professional's next move remains Chapter 31's systematic aggregation, not this anecdotal parade.

🌎

Real World: RBF SVMs were the default tabular classifier in industry for a decade (roughly 2005–2015) — protein classification, image features, credit scoring — before gradient-boosted trees took the crown. They remain the tool of choice for medium-size, high-dimensional problems (think 10k rows × 5k features) where forests need more data and deep nets need far more: the kernel trick's infinite features shine exactly when features are already numerous.

8. Choosing Kernels & the Road Ahead

Situation	Reach for	Why
Large data, suspect linearity	`linear` (or LinearSVC)	Fast, honest baseline; if it suffices, done
Nonlinear, medium size, unclear shape	`rbf` (default)	Local bumps fit almost anything; one main dial (γ)
Domain suggests polynomial structure (physics, interactions)	`poly`	Global monomial shape matches the theory
Deploying beyond prototype	Grid search over C × γ (Ch 31)	Dials interact; systematic beats manual

The standing obligations: scaling stays mandatory (kernels compute distances — Chapter 13's catastrophe applies unchanged), probabilities still need calibration, and training cost still grows steeply with rows. What the kernel trick added is the lesson this Part will keep reusing: change the space, not the algorithm. Next chapter swaps geometry for probability entirely — Naive Bayes classifies not by streets or votes but by (a famously naive simplification of) Bayes' theorem, trains in one pass, and asks a question no model has asked yet: given the features, which class is most probable?

🛠 Hands-On: Lift, Menu, DialIntermediate

Your team debates "poly or RBF?" over lunch. Settle it with evidence: work the lifting example by hand, run the full kernel menu and the γ sweep, and deliver a one-page kernel-choice recommendation with the boundary customer's verdict attached.

Steps

Hand-verify Figure 14.1: compute z = x² for x = −2, 0, +2 and show z = 2 separates; explain why no threshold in x works.
Expand (a·b + 1)² by hand and confirm it equals the inner product of (1, √2·a, a²) with (1, √2·b, b²) — the kernel trick, checked by algebra.
Run the kernel menu (linear, poly, poly-d2, rbf, sigmoid); tabulate test/train accuracy and SV counts. Which result surprises, and why?
Run the γ sweep {0.01, 0.1, 0.5, 1, 10, 100}; identify the three regimes and mark the plateau.
Report decision_function for (30, ₹87k), (30, ₹150k), (45, ₹87k) under RBF; compare (30, ₹150k) with the linear SVM's +0.21.
Write the recommendation: kernel, γ guidance, scaling reminder, and what Chapter 31 would add before deployment.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler()
Xs_tr, Xs_te = sc.fit_transform(X_tr), sc.transform(X_te)

# Step 2, the algebra check
a, b = 3, 5
print((a*b + 1)**2, '==', 1 + 2*a*b + a*a*b*b)          # 256 == 256

for k in ['linear', 'poly', 'rbf', 'sigmoid']:
    m = SVC(kernel=k).fit(Xs_tr, y_tr)
    print('%-8s test=%.3f train=%.3f SVs=%d' %
          (k, m.score(Xs_te, y_te), m.score(Xs_tr, y_tr), m.support_.size))

for g in [0.01, 0.1, 0.5, 1, 10, 100]:
    m = SVC(kernel='rbf', gamma=g).fit(Xs_tr, y_tr)
    print('gamma=%-5g test=%.3f train=%.3f SVs=%d' %
          (g, m.score(Xs_te, y_te), m.score(Xs_tr, y_tr), m.support_.size))

rbf = SVC(kernel='rbf').fit(Xs_tr, y_tr)
for cust in [[30, 87000], [30, 150000], [45, 87000]]:
    print(cust, rbf.decision_function(sc.transform([cust]))[0].round(2))

Expected results (real):

Step 1: z-values 4, 0, 4; threshold z = 2 separates. In x, every cut puts one wing of A with B — "outside vs inside" needs a second dimension.
Step 2: (ab+1)² = a²b² + 2ab + 1 = the inner product of the lifted vectors — the kernel computes the lifted geometry from raw inputs (e.g. a=3, b=5: 256 = 256).
Step 3: linear 0.90/128 SVs · poly 0.86/125 · poly-d2 0.74/154 (the surprise: the "natural" curved kernel collapses) · rbf 0.93/88 · sigmoid 0.86/125.
Step 4: flat 0.89 at γ=0.01 (186 SVs) → plateau 0.93 at γ=0.5–1 (88 SVs; 'scale' = 0.5) → crash 0.82 at γ=100 (train 0.973, 271 SVs — memorisation).
Step 5: −1.03 / +1.13 / +0.34 — the RBF street commits to the boundary customer five times more firmly than the linear street's +0.21.
Step 6: RBF at the γ plateau (or 'scale'), scaled inputs, C co-tuned on a validation grid (Ch 31) before deployment; poly-d2's collapse documented as the reason kernel choice is a hypothesis, not a setting.

🎯 Key Takeaways

Lifting rescues separation: outer-at-±2 vs inner-at-0 is hopeless in x, trivial in z = x² — curved regions are straight streets in disguise.

The SVM needs only pairwise inner products — so a kernel supplies the lifted geometry without building any columns. RBF lifts to infinite dimensions, for free.

Kernel choice is a hypothesis: RBF 0.93 vs poly-d2's 0.74 on the same data — the "obvious" curved kernel can be the worst one. Verify, don't assume.

γ is locality: too global flattens (0.89, 186 SVs), the plateau traces the region (0.93, 88), too local memorises (train 0.97 / test 0.82, 271 SVs).

The curved street ties K-NN (0.93) while storing 88 support vectors instead of 300 rows — flexibility and economy in one trained object.

Everything from Ch 13 carries: C, hinge loss, convexity, scaling-mandatory. The lesson that generalises: change the space, not the algorithm.