Part II — Regression

Chapter 7 · Support Vector Regression (SVR)

OLS punishes every residual, however tiny. Support Vector Regression makes a different deal: small errors are free. Fitting a tube around the curve instead of threading through every point buys robustness, sparsity and — via kernels — enormous flexibility. Same salary data as Chapter 6, a completely different philosophy, and a real parameter study where defaults are exposed for what they are: starting points.

📖 Chapter 7 of 33 ⏱ ~90 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Explain SVR's ε-insensitive tube and how it changes the definition of "best fit" compared to OLS
Identify support vectors, explain the solution's sparsity, and interpret the C penalty and slack variables
Demonstrate — with real numbers — why SVR refuses unscaled data while Chapters 4–6 never needed scaling
Give the RBF kernel's formula and the working intuition for its γ parameter
Predict how C, ε and γ move a fit, and verify against the real parameter study on Position_Salaries.csv
Compare SVR and polynomial regression on the same data — fit, extrapolation, interpretability — and choose between them defensibly

1. A Different Definition of "Best Fit"

Chapters 4–6 minimised the sum of squared residuals — every point votes, and points far from the line vote loudest. Support Vector Regression (SVR) rewrites the contract. Build a tube of half-width ε around the fitted curve, and residuals inside the tube cost nothing. The optimisation asks for the flattest tube that captures most points, penalising only the points that escape it:

minimize ½‖w‖² + C · Σ (ξ_i + ξ_i*)

½‖w‖² — flatness of the curve (small weights = gentle function)
ε (epsilon) — the tube's half-width: errors within ±ε are free
ξ, ξ* — slack: how far each escaping point sits above/below the tube
C — the price per unit of slack (Section 6 gives each dial a real-number story)

Figure 7.1 — The ε-insensitive tube: hollow points inside are free; filled points on the boundary are support vectors; the red point pays slack.

Why would anyone want to ignore small errors? Because chasing them is how models memorise noise. The tube declares a tolerance band — "errors this small are indistinguishable from noise" — and spends all its flexibility on the shape that remains. It is bias-variance Chapter 6 with the dial relabelled: wider tube = more bias, more tolerance; narrower tube = more variance, more chasing.

2. Inside the ε-Tube: Support Vectors & Slack

The tube does more than tolerate errors — it performs feature selection on the data itself. At the optimum, only the points sitting on or outside the tube boundary determine the curve. These are the support vectors. Every point strictly inside the tube could be deleted and the fit would not move by one decimal — its gradient contribution is literally zero.

The fitted model is a weighted sum of kernel similarities to the support vectors alone (Section 4 gives the kernel):

ŷ(x) = Σ_{i ∈ SV} α_i · K(x_i, x) + b

The sum runs over support vectors only — on this chapter's data, 5 of the 10 points carry the entire model. This sparsity is SVR's signature: compact models, cheap predictions, and a natural robustness to noise the tube already absorbs.

Points that escape the tube are not discarded — they are fined. The slack variables ξ measure each escape distance, and C sets the fine per unit. Small C: escapes are cheap, the tube stays wide and placid. Large C: escapes are expensive, the tube bends to keep points close. Section 6 watches both happen in real numbers.

📝

Note — the classification family tree: SVR is regression's sibling of the SVM you will meet in Chapters 13–14 (margins between classes instead of tubes around a curve). Both share the support-vector machinery, the kernel trick, and the C penalty — learning the vocabulary here makes half of Chapter 13 a rerun.

3. Why SVR Refuses Unscaled Data

Chapters 4–6 never scaled anything — OLS solves the normal equations exactly, with no distances and no gradients. SVR is our first geometric regressor, and geometry has units. Two components break immediately on raw data:

The kernel computes distances in x-space. Levels 1–10 are small numbers; had we a second feature in rupees, it would dominate every distance. (Chapter 3's table said SVM-family models must be scaled — this is why.)
ε is a distance in y-space. The default ε = 0.1 is meaningless on salaries of 45,000–1,000,000 — it demands near-perfection, and the model flatlines trying, or drowns the solver in scale. Standardise y (mean 0, SD 1) and ε = 0.1 becomes a principled tolerance: a tenth of a standard deviation, ≈ ±₹28,401 on this dataset (SD of salary = ₹284,011).

The proof is one line of code away (real output):

Pipeline	R²	Prediction at 6.5	Verdict
Scale X and y (source)	0.7516	₹170,370	The working pipeline
Scaled X, raw y	−0.177	₹130,002	Worse than predicting the mean — the tube demands impossible precision

A negative R²: the model is beaten by a constant guess. No error message, no warning — just a silent catastrophe, exactly the kind Chapter 3's ritual (audit, then transform deliberately) exists to prevent.

💡

Pro Tip: Chapter 6's polynomial never scaled because OLS is algebraic; SVR must scale because kernels are geometric. That one sentence is the exam answer to "when do I need feature scaling?" — and Chapter 3's table now has two filled-in columns of evidence behind it.

4. The Kernel Trick — Preview

Chapter 6 closed on a structural insight: flexibility comes from features. A degree-4 polynomial is a 4-column feature expansion. The natural continuation — degree 20, degree 100, with interaction terms across every feature — explodes computationally: the columns become unbuildable.

The kernel trick is the escape: SVR's optimisation never actually needs the transformed features — it only ever needs similarities between pairs of points. Supply those similarities directly, and you have implicitly fitted in an enormous (for the RBF kernel, effectively infinite-dimensional) feature space without building a single column:

K(x, x′) = exp( −γ · ‖x − x′‖² )

The RBF (Gaussian) kernel: similarity 1 for identical points, decaying smoothly with distance. Each support vector contributes a soft local bump; the prediction is a weighted blend of bumps. γ (gamma) sets each bump's width: small γ = wide bumps = global, gentle curves; large γ = narrow bumps = spiky, local, sample-chasing behaviour.

Figure 7.2 — γ is the bump-width dial on the bias-variance machine.

This section is deliberately a preview: the full treatment — why the trick is legal, the kernel menu (linear, polynomial, sigmoid, RBF), and choosing among them — is Chapter 14's job, where classification makes the geometry vivid. Here you need only the working model: RBF SVR fits a sum of local bumps whose width γ you control, in a feature space you never build.

📝

Note — a real quirk worth knowing: sklearn's default gamma='scale' computes γ = 1/(n_features × Var(X)). With one standardised feature, that is exactly γ = 1.0 — the "default" and "γ = 1" fits on this dataset are the same fit, a coincidence of standardisation, not a law.

📋 Stable content — Reviewed: August 2026

5. SVR in Python

The source pipeline, annotated. The one structural novelty versus Chapters 4–6: two scalers — X and y are different objects needing different transformations, and predictions must be mapped back to rupees.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:-1].values.astype(float)   # Level
y = dataset.iloc[:, -1].values.astype(float).reshape(-1, 1)   # Salary as a COLUMN

# Feature scaling — both sides (Section 3's evidence)
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
sc_y = StandardScaler()
X = sc_X.fit_transform(X)
y = sc_y.fit_transform(y)

# Fit RBF SVR (defaults: C=1.0, epsilon=0.1, gamma='scale')
from sklearn.svm import SVR
regressor = SVR(kernel='rbf')
regressor.fit(X, y.ravel())        # SVR wants a 1-D target
print('support vectors:', regressor.support_.size)          # 5 of 10

# Predict a new observation — three steps: scale x, predict, unscale y
scaled_pred = regressor.predict(sc_X.transform([[6.5]]))
salary = sc_y.inverse_transform(scaled_pred.reshape(-1, 1))
print(f'salary at 6.5: ₹{salary[0, 0]:,.0f}')               # ₹170,370

# Smooth curve for plotting — in original units
X_grid = np.arange(min(sc_X.inverse_transform(X)),
                   max(sc_X.inverse_transform(X)), 0.1).reshape(-1, 1)
plt.scatter(sc_X.inverse_transform(X), sc_y.inverse_transform(y), color='red')
plt.plot(X_grid, sc_y.inverse_transform(
    regressor.predict(sc_X.transform(X_grid)).reshape(-1, 1)), color='blue')
plt.title('Truth or Bluff (SVR)'); plt.xlabel('Level'); plt.ylabel('Salary')
plt.show()

Listing 7.1 — The complete SVR pipeline (source: support_vector_regression.py).

💡

Pro Tip — the reshape dance: StandardScaler insists on 2-D input (hence reshape(-1, 1) for y), while SVR.fit wants a 1-D target (hence .ravel()), and inverse_transform wants 2-D again. Three shapes, one variable — the pipeline runs fine but only after you humour every function's preference. Treat the listing as the canonical choreography.

6. Reading the Fit: C, ε, γ — Real Numbers

Every SVR parameter is a bias-variance dial. Here is the real parameter study — one setting changed at a time from the default (C=1, ε=0.1, γ='scale'=1), same ten rows, R² measured in salary units:

Setting	R²	Pred @6.5	SVs	What happened
Default (source)	0.7516	₹170,370	5	Honest but mediocre — the curve sags under the CEO point
C = 10	0.9886	₹163,534	6	Escapes priced higher — the tube chases the big salaries
C = 100	0.9928	₹193,873	5	Near-total obedience; watch @6.5 drift up — variance creeping in
γ = 0.1	0.4641	₹204,151	5	Bumps too wide — one gentle swell, underfits the bend
γ = 10	0.7474	₹199,019	8	Bumps too narrow — wiggles toward individual points
ε = 0.5	0.6137	₹304,589	5	Tube ±₹142k — half the data rides free, fit flattens
ε = 1.0	0.0912	₹448,916	4	Tube ±1 SD swallows nearly everything — the model nearly flatlines
ε = 0.01	0.7565	₹166,614	8	Tube ±₹2.8k — more points forced onto the boundary
C=10, ε=0.01	0.9904	₹174,144	9	Strict and expensive — 9 of 10 points become support vectors

Four readings worth engraving:

C is the obedience dial. R² 0.75 → 0.99 as C goes 1 → 10: the penalty now outweighs flatness, and the tube stretches to hold the extreme salaries. Push to 100 and the @6.5 prediction starts wandering (₹193,873) — obedience to the sample, i.e. variance, Chapter 6's word.
ε is the tolerance dial — in rupees. ε=0.1 means "±₹28,401 is free"; widen to ±₹142k (0.5) and the fit visibly sags; at ±1 SD (1.0) almost every point rides free and the model collapses toward a near-flat average (R² 0.09). The tube is not a technicality — it IS the model's philosophy.
γ is the locality dial. Both directions lose here (0.46 and 0.75): with ten points, the default width happens to sit in the sweet spot — more evidence that dials need evidence, not instinct.
SV count is the sparsity gauge. 5 support vectors (default) → 9 (strict tube + high C). As tolerance shrinks and obedience rises, more points become load-bearing — the model concentrates, and generalisation depends ever more heavily on those few rows.

⚠

Warning — the default is a starting point, not a verdict: the source code ships defaults and never tunes, so its R² of 0.75 understates the method. The professional loop — try, measure, adjust — is exactly this section's table, and Chapter 31's grid search is its industrial version. Never report an untuned SVR as SVR's best.

7. SVR vs Polynomial — Two Philosophies

Same dataset, same question (₹160k at level 6.5?), two very different machines. Both answers below are real:

	Poly degree 4 (Ch 6)	SVR default	SVR C=10
R² (in-sample)	0.997	0.752	0.989
Prediction @6.5	₹158,862	₹170,370	₹163,534
Prediction @12 (extrapolation)	₹3,035,985	₹461,423	₹1,174,312
Model size	5 coefficients	5 support vectors	6 support vectors
Feature scaling	Not needed (exact solver)	Mandatory, X and y
Interpretability	Equation you can read	Weights on kernel bumps — opaque
Flexibility mechanism	Explicit feature columns	Kernel trick — implicit features, no columns

Three verdicts. On interpolation, both lands near the claim once tuned — polynomial ₹158,862, SVR ₹163,534 against ₹160,000: two philosophies, one answer, which is how you know the data speaks. On extrapolation they part ways completely: the polynomial detonates (its x⁴ term owns level 12) while RBF SVR — a blend of local bumps — decays toward a bounded prediction: still not trustworthy beyond the range, but not absurd. On interpretability the polynomial wins outright; a CFO can hear "the curve bends like x⁴", while SVR offers weights on invisible bumps.

🌎

Real World: RBF SVR's quiet superpower is the tube's noise immunity. With 10,000 points of jittery sensor data, the polynomial family must keep raising degree to chase the bend while the tube shrugs off the jitter for free. The salary grid — smooth, tiny, clean — is actually polynomial territory; we use it here to learn the machinery, not because SVR is the natural champion of ten tidy points.

8. Strengths, Limits & the Road Ahead

Strengths	Limitations
Tube = built-in noise tolerance (robustness by design)	Scaling of X and y is mandatory — silently fatal if skipped
Kernel trick: enormous flexibility, no feature construction	Three coupled hyperparameters (C, ε, γ) — needs systematic tuning
Sparse solutions — only support vectors matter	Opaque: no equation to read to a stakeholder
Excellent on medium-size, nonlinear, noisy problems	Training cost grows steeply with data size; struggles at millions of rows

With SVR, Part II's toolkit now spans three philosophies: exact equations (Chapters 4–5), explicit feature bending (Chapter 6), and implicit kernel bending here. Chapter 8 takes a fourth road entirely — decision trees: axis-aligned splits, no scaling, no equation, a model you can literally draw as a flowchart. The variety is the point: by Chapter 10 you will benchmark all of them under one honest metric system.

🛠 Hands-On: Tune the TubeIntermediate

Your SVR colleague shipped the default model and R² = 0.75. Management asks: "Can this model do better, and what does each knob actually do?" Produce the evidence — parameter study, sparsity counts, and a recommendation.

Steps

Run the Section 5 pipeline; record default R², @6.5, and the support-vector count.
Vary C over {1, 10, 100} (others default). Which direction fits harder? Where does @6.5 start drifting?
Vary ε over {0.01, 0.1, 0.5, 1.0}. Convert each to rupees (ε × SD of salary). What does ε = 1.0 do to the fit, and why?
Vary γ over {0.1, 1, 10}. Describe each curve's personality (global swell vs local wiggles).
Break it deliberately: fit on raw (unscaled) y. What R² do you get, and what does the model predict at 6.5?
Recommend a final configuration with one paragraph: fit, philosophy, and the Ch 6 comparison.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
from sklearn.metrics import r2_score

ds = pd.read_csv('Position_Salaries.csv')
X = ds.iloc[:, 1:-1].values.astype(float)
yf = ds.iloc[:, -1].values.astype(float)
sc_X, sc_y = StandardScaler(), StandardScaler()
Xs = sc_X.fit_transform(X)
ys = sc_y.fit_transform(yf.reshape(-1, 1)).ravel()

def run(name, **kw):
    r = SVR(kernel='rbf', **kw).fit(Xs, ys)
    pred = sc_y.inverse_transform(r.predict(Xs).reshape(-1, 1)).ravel()
    at65 = sc_y.inverse_transform(
        r.predict(sc_X.transform([[6.5]])).reshape(-1, 1)).ravel()[0]
    print(f'{name:24s} R2={r2_score(yf, pred):8.4f}  @6.5={at65:9,.0f}  SV={r.support_.size}')

run('default')
for C in [1, 10, 100]:   run(f'C={C}', C=C)
for e in [0.01, 0.5, 1.0]: run(f'eps={e}', epsilon=e)
for g in [0.1, 10]:      run(f'gamma={g}', gamma=g)

# the deliberate break
r_bad = SVR(kernel='rbf').fit(Xs, yf)
print('raw y : R2=%.4f  @6.5=%.0f' % (r2_score(yf, r_bad.predict(Xs)),
                                      r_bad.predict(sc_X.transform([[6.5]]))[0]))

Expected results (real):

C: R² 0.7516 → 0.9886 (C=10) → 0.9928 (C=100); @6.5 drifts 170,370 → 163,534 → 193,873 — the C=100 wobble is variance arriving.
ε in rupees (SD = 284,011): 0.01→±₹2.8k, 0.1→±₹28.4k, 0.5→±₹142k, 1.0→±₹284k. R²: 0.7565 / 0.7516 / 0.6137 / 0.0912 — at ±1 SD nearly everything rides free and the model almost flatlines.
γ: 0.1 → R² 0.4641 (one gentle swell — underfit); 10 → R² 0.7474 with 8 SVs (local wiggles chasing points).
Broken pipeline: raw-y fit gives R² = −0.177 and predicts ₹130,002 at 6.5 — a near-mean guess wearing a model's clothes.
Recommendation: C=10 (R² 0.989, @6.5 ₹163,534 — within ₹3.5k of the candidate's claim, matching Chapter 6's polynomial answer of ₹158,862 through a completely different philosophy). Note honestly: in-sample on 10 points; extrapolation stays forbidden.

🎯 Key Takeaways

SVR redefines "best fit": errors within the ε-tube are free, and only escapees are priced — flatness plus tolerance instead of squared-residual bookkeeping.

Only support vectors carry the model. On this data: 5 of 10 points; delete the rest and the curve doesn't move.

Kernels are geometric, so scaling is mandatory — for X (distances) and y (ε is a distance too). Raw-y fit: R² −0.18, silently worse than the mean.

The RBF kernel fits sums of local bumps in an implicit infinite-dimensional space — Chapter 6's feature flexibility without the columns.

C = obedience, ε = tolerance (in target units), γ = locality — all three are bias-variance dials, and the real study moved R² from 0.09 to 0.99.

Defaults are starting points: the source's untuned 0.75 became 0.99 at C=10. Tuning with evidence — not instinct — is the professional reflex (systematised in Chapter 31).