Part II — Regression

Chapter 6 · Polynomial Regression & the Bias-Variance Tradeoff

Some relationships bend. Polynomial regression bends them while keeping everything from Chapters 4–5 — ordinary least squares, the assumptions, the diagnostics — by transforming features instead of changing the maths. That one trick unlocks the course's central idea: the bias-variance tradeoff, the dial every model from here to Chapter 33 turns. This chapter proves the tradeoff with a real experiment, not a metaphor.

📖 Chapter 6 of 33 ⏱ ~90 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Explain why "linear regression" means linear-in-coefficients, not linear-in-x — and why that distinction powers everything in this chapter
Build polynomial features by hand and fit them with the same OLS as Chapters 4–5
Fit degrees 1–4 on Position_Salaries.csv and read what each degree does to fit, predictions and coefficient stability
State the bias-variance decomposition precisely: expected test error = Bias² + Variance + irreducible noise
Demonstrate the U-shaped test-error curve experimentally, and locate underfitting and overfitting on it
Choose a degree defensibly — and predict where polynomial models fail (extrapolation)

1. When Straight Lines Fail

The running scenario is a negotiation. A candidate says: "My last title was Partner, but between Partner and Senior Partner — call it level 6.5 — and I earned ₹160,000." HR has ten salary levels and wants to know: truth or bluff? Position_Salaries.csv holds the company's grid:

Level	1	2	3	4	5	6	7	8	9	10
Salary (₹)	45k	50k	60k	80k	110k	150k	200k	300k	500k	1,000k

Ten rows, one feature, one target — Chapter 4's territory. But plot it and the problem announces itself: salaries creep at low levels and rocket at high ones. A straight line through these points misses low and high while cutting through the middle — the relationship bends.

Fit the line anyway (real output) and watch it fail:

Model	Equation (real fit)	R²	Prediction at 6.5
Degree 1 (line)	ŷ = −195,333 + 80,879·x	0.669	₹330,379
The candidate's claim	—	—	₹160,000

R² = 0.669 — a third of the variation unexplained — and the ₹330,379 prediction is double the candidate's claim, dragged up by the steep right end. The line is the wrong family of curve. Chapter 1 named this failure: underfitting. This chapter fixes it — and then, honestly, shows how the fix overshoots.

Figure 6.1 — Ten salary levels: the straight line misses both ends; the degree-4 curve tracks the bend. (Levels at 6.5 — the negotiation question — marked.)

2. Polynomial Feature Expansion

Here is the trick that keeps this chapter inside Chapters 4–5 rather than starting new mathematics. "Linear" regression was never a promise about x — it is a promise about the coefficients. If we manufacture new columns — x², x³, x⁴ — the model stays linear in its parameters, and OLS works unchanged:

ŷ = b₀ + b₁x + b₂x² + … + b_dx^d

Linear in the b's — nonlinear in x. Once the columns exist, this is multiple linear regression on transformed features: same normal equations, same R², same residual diagnostics, same "holding others constant" reading (here, the others are powers of the same x, so b₁ reads as the effect of x near x = 0 — interpretation shifts, the machinery does not).

The feature factory in sklearn is one class:

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=4, include_bias=False)
X_poly = poly.fit_transform(X)      # column x becomes [x, x2, x3, x4]

Listing 6.1 — Degree-4 expansion: four columns from one.

2.1 Worked by hand — three points, one parabola

📝

Read this first — why degree 2, when Listing 6.1 says degree 4? The degree in PolynomialFeatures(degree=…) is the d in the formula above — the highest power of x the factory builds. This worked example deliberately uses degree 2 (a parabola): three points and three unknowns give clean, hand-solvable arithmetic. The mechanism you are about to perform — points become equations, equations become coefficients — is identical for every degree. Learn it at degree 2; the computer runs the same recipe at degree 4 on the salary data.

Start with the counting rule that ties degree to everything else. A degree-d polynomial has d + 1 coefficients to find, so it needs d + 1 points to be pinned down exactly:

Degree d	Model	Coefficients	Columns PolynomialFeatures builds	Points it can pass through exactly
1	line: b₀ + b₁x	2	[x]	2
2	parabola: b₀ + b₁x + b₂x²	3	[x, x²]	3
4 (Listing 6.1)	quartic: b₀ + b₁x + … + b₄x⁴	5	[x, x², x³, x⁴] — b₀ comes separately from LinearRegression's intercept	5

So three points determine one parabola exactly. Take (1, 2), (2, 5), (3, 10) and fit ŷ = b₀ + b₁x + b₂x². The recipe: each point plugs its own x into the model and demands the prediction match its y — one equation per point:

point (1, 2):   b₀ + b₁·1 + b₂·1² = 2    → b₀ + b₁ + b₂ = 2
point (2, 5):   b₀ + b₁·2 + b₂·2² = 5    → b₀ + 2b₁ + 4b₂ = 5
point (3, 10): b₀ + b₁·3 + b₂·3² = 10   → b₀ + 3b₁ + 9b₂ = 10

Three equations, three unknowns (b₀, b₁, b₂) — solvable by elimination. Watch where each column's number comes from: the coefficient of b₂ is always the point's x squared (1, 4, 9 — that is 1², 2², 3²).

Elimination, one subtraction at a time — watch which terms cancel:

Eq2 − Eq1: (b₀−b₀) + (2b₁−b₁) + (4b₂−b₂) = 5−2 → b₁ + 3b₂ = 3
Eq3 − Eq2: (b₀−b₀) + (3b₁−2b₁) + (9b₂−4b₂) = 10−5 → b₁ + 5b₂ = 5

then (second) − (first): (b₁+5b₂) − (b₁+3b₂) = 5−3 → 2b₂ = 2 → b₂ = 1

The first subtractions kill b₀ (every equation contains it); the final subtraction kills b₁. Each step eliminates one unknown, leaving a smaller system — Gauss's elimination, the algorithm inside OLS.

Back-substitute up the chain — each solved coefficient feeds the earlier equations:

b₂ = 1 into b₁ + 3b₂ = 3: b₁ + 3(1) = 3 → b₁ = 0
b₁ = 0, b₂ = 1 into Eq1: b₀ + 0 + 1 = 2 → b₀ = 1

ŷ = 1 + 0·x + 1·x²

Verify all three points, not just one: x=1 → 1+0+1 = 2 ✓ · x=2 → 1+0+4 = 5 ✓ · x=3 → 1+0+9 = 10 ✓. The parabola threads all three exactly — that is what "three points determine it" means.

Now reread Listing 6.1 with new eyes. PolynomialFeatures(degree=2) would build exactly our two columns, [x, x²], and LinearRegression().fit() would solve for (b₀, b₁, b₂) via the normal equations — elimination's industrial-scale cousin, arriving at the same 1, 0, 1. Change the listing to degree=4 (as it is) and the same two lines hunt five coefficients (b₀…b₄) on ten salary points: more coefficients than any three points could pin down, so among all possible quartics, least squares selects the best-fitting one — precisely how Chapter 4 selected among all lines.

And notice what degree bought: a line (2 coefficients) cannot pass through three arbitrary points; the parabola (3 coefficients) is guaranteed to. More degree = more coefficients = more bending budget. That is the bias-variance tradeoff in embryo — and the counting rule behind Section 3's whole degree table (degree 9's ten coefficients against ten salary points = memorisation).

📝

Note: Chapter 5's assumptions all still apply — to the expanded feature set. Multicollinearity deserves special mention: x and x² are correlated by construction, so their individual coefficients blur (high VIFs are expected and tolerated); what matters is the curve, not the decomposition of it. This is also why polynomial coefficients swing wildly at high degrees — Section 5's headline.

3. Fitting Degrees 1–4 on Real Data

Fit every degree on the ten salary rows (real output, whole dataset — see the honesty note below):

Degree	Parameters	R² (in-sample)	Prediction at 6.5	Prediction at 12 (extrapolation)
1	2	0.669	₹330,379	₹775,212
2	3	0.916	₹189,498	₹1,435,894
3	4	0.981	₹133,259	₹2,175,030
4	5	0.997	₹158,862	₹3,035,985
5	6	0.9994	₹170,446	₹3,592,972
9	10	0.9998	₹176,785	₹3,521,725

Three readings before the theory arrives:

Degree 1 underfits: R² 0.669, prediction ₹330k — double the candidate's claim.
Degree 4 lands near the claim: ₹158,862 against the candidate's ₹160,000 — and the curve (Figure 6.1) visibly hugs the bend. This is the source code's model of choice.
R² keeps climbing past degree 4 — 0.997 → 0.9994 → 0.9998 — without any evidence the higher curves are better. More parameters fit noise as happily as signal; Chapter 5's "R² never falls" trap, amplified.

⚠

Warning — the honesty note: these R² values are in-sample. The source code fits the whole dataset with no train/test split — defensible with ten rows (a split would leave too few to learn or test from), but it means nothing here measures generalisation. In-sample fit mechanically improves with degree; only held-out data (Section 7's cross-validation) can rank degrees honestly.

3.1 The degree-2 curve deserves a look

Degree 2's mid-range is instructive: its fitted parabola (ŷ = 232,167 − 132,871x + 19,432x²) predicts ₹53,606 at level 5 — against an actual ₹110,000. A single quadratic term captures the eventual upturn but dips through the middle. R² 0.916 looks respectable while the curve is visibly the wrong shape — Chapter 4's lesson that fit statistics never replace looking at the curve, again.

4. The Bias-Variance Tradeoff — Formal

Chapter 1 introduced overfitting and underfitting with pictures. With polynomial degrees in hand, both can be made precise. Fix a test point x, draw many training sets from the world, fit a model on each, and ask how the predictions ŷ scatter around the true value y. The expected squared error decomposes exactly:

E[ (y − ŷ)² ] = Bias²[ŷ] + Var[ŷ] + σ²

Bias² — how far the average prediction sits from the truth: systematic error from using the wrong model family (a line for a bending world)
Var — how much predictions wander across training sets: sensitivity to the particular sample (a 9th-degree polynomial reinvents itself on every draw)
σ² — irreducible noise in y itself. No model, however clever, predicts the unpredictable part.

Figure 6.2 — Left: the four bias-variance regimes as targets (centre = truth). Right: training error falls monotonically with complexity while test error bends back up — the U that governs model choice.

4.1 The tradeoff, demonstrated — not asserted

Metaphors are cheap; here is a real experiment. The true function is a known parabola, y = 0.8x² − 2x + 3, measured with Gaussian noise (σ = 1.5). We draw 300 random training sets of 20 points each, fit every degree 1–9, and score each on a fresh 50-point test set. The averaged results (real run, seed 42):

Degree	1	2	3	4	5	6	7	9
Train MSE	3.92	1.88	1.77	1.65	1.54	1.47	1.47	1.57
Test MSE	5.06	2.69	2.92	4.36	9.15	78.0	137.9	488.6

Read the two rows against the theory. Training error falls almost monotonically — complexity always fits the sample better. Test error is U-shaped: it bottoms exactly at degree 2, the true complexity, then climbs — gently at degree 3–4 (variance beginning to bite), catastrophically by degree 6+ (predictions swinging hundreds of units on the next sample's whim). Degree 1's gap (train 3.92 vs its own test 5.06, both poor) is bias; degree 9's gap (train 1.57, test 488.6) is variance. The decomposition is not a diagram — it is this table.

💡

Pro Tip: Notice the sweet spot sits at the data-generating complexity, not at the minimum training error. In real problems the truth is hidden, which is why degree choice needs held-out evidence — Section 7's cross-validation — never in-sample R².

5. Overfitting in Action

Return to the salary data and push the degree. Three signatures of overfitting appear — each one a habit you can audit in any model, not just polynomials.

Signature 1 — near-perfect in-sample fit from parameter count alone

A degree-9 polynomial plus intercept has exactly 10 parameters for 10 data points: enough to pass (numerically) through every observation. It "scores" R² = 0.9998. That is not learning the salary policy — it is interpolating it, noise included. Ask the degree-9 model about level 6.5 and its ₹176,785 answer happens to look sane; ask it about a level nobody occupies and the mask slips.

Signature 2 — coefficient instability

The degree-4 fit (real coefficients):

ŷ = 184,167 − 211,002·x + 94,765·x² − 15,463·x³ + 890·x⁴

Alternating signs and huge magnitudes — a −₹211k "effect" of level cancelled by a +₹95k square, itself cancelled by a −₹15k cube. The curve is excellent; the decomposition is an accounting fiction, highly sensitive to the exact ten rows (and to collinearity between powers, per Section 2's note). When coefficients fight each other in cancelling waves, the model has spent its bending budget on the sample's particulars.

Signature 3 — extrapolation detonation

Section 3's last column is the smoking gun. At level 12 — two steps beyond the data — the degree-4 model predicts ₹3,036,000, degree 5 says ₹3.6M, and the candidate is presumably delighted. Polynomials are captive to their highest-degree term: past the observed range, x⁴ owns the curve and values soar or plummet on its sign. Chapter 4 flagged extrapolation as risky for lines; for polynomials it is meaningless.

⚠

Warning: An overfit model usually looks great on every question you can check — because you are checking it on the data it memorised. Its failures live where you cannot check: new samples and out-of-range inputs. That asymmetry is the entire practical danger.

🌎

Real World — an honest twist: leave-one-out cross-validation on this particular dataset actually keeps improving up to degree 5 (LOO mean R²: −1.79 at degree 1, 0.06 at 2, 0.79 at 3, 0.92 at 4, 0.98 at 5). The salary grid is a genuinely smooth curve with no noise, so extra flexibility buys real fit — overfitting's bill here arrives out of range (the extrapolation column) rather than in-sample. On noisy data — the Section 4 experiment — the same flexibility is ruinous. Diagnosis always depends on how much noise your data carries; never apply "degree 4 overfits" as a slogan.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

The complete source workflow, modernised and annotated. Note the shape of the pipeline: PolynomialFeatures transforms, LinearRegression does the rest — two objects, one model.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:-1].values     # Level only — column 0 is the job title (text)
y = dataset.iloc[:, -1].values       # Salary

# Model 1: plain linear (the underfit baseline)
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression().fit(X, y)

# Model 2: polynomial, degree 4
from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree=4, include_bias=False)
X_poly = poly_reg.fit_transform(X)               # [x, x2, x3, x4]
lin_reg_2 = LinearRegression().fit(X_poly, y)

print('linear R2 : %.3f' % lin_reg.score(X, y))          # 0.669
print('poly   R2 : %.3f' % lin_reg_2.score(X_poly, y))   # 0.997

# Smooth curve plot: evaluate on a fine grid (plotting only — the model is unchanged)
X_grid = np.arange(min(X), max(X), 0.1).reshape(-1, 1)
plt.scatter(X, y, color='red')
plt.plot(X_grid, lin_reg_2.predict(poly_reg.transform(X_grid)), color='blue')
plt.title('Truth or Bluff (Polynomial Regression)')
plt.xlabel('Position level'); plt.ylabel('Salary')
plt.show()

# Predicting a new observation — transform FIRST, then predict
print(lin_reg.predict([[6.5]]))                                  # [330378.9]
print(lin_reg_2.predict(poly_reg.transform([[6.5]])))            # [158866.2]

Listing 6.2 — The full degree-4 workflow (source: polynomial_regression.py).

💡

Pro Tip — the two classic bugs: (1) The source calls poly_reg.fit_transform([[6.5]]) on new data — refitting the transformer on one value. It happens to work here (the transform is deterministic, stateless), but the disciplined pattern is poly.transform(...): Chapter 3's rule — fit on training data, transform new data — applies to feature factories too. (2) Set include_bias=False: with it, PolynomialFeatures prepends a column of 1s and LinearRegression adds its own intercept — a duplicate that makes the design rank-deficient (Chapter 5's trap wearing a new coat).

7. Choosing the Degree

By now the diagnosis rules are clear. In-sample R² cannot choose a degree — it monotonically rewards complexity (Sections 3–4). The honest instruments, in order of authority:

Domain sense. Should the effect accelerate, saturate, or grow steadily? Salary grids bend upward; drug dose-responses saturate. If physics/business says "bend", start at degree 2–3.
Held-out evidence. Cross-validation — Chapter 31's machinery — scores each degree on data it never trained on. On the synthetic experiment it located degree 2 exactly; on the salary grid it refined the degree-4 choice. Small datasets: leave-one-out (Section 5's Real World).
Parsimony. Between degrees with statistically indistinguishable held-out scores, take the smaller — fewer parameters, stabler coefficients, saner extrapolation (Chapter 5's adjusted-R² instinct, applied to degrees).
Diagnostics. Residual-vs-fitted still applies: a curve left in the residuals says "more bend needed" — the polynomial cousin of Chapter 4's LINE check.

💡

Pro Tip: Degrees above 3–4 on raw features are rarely worth the trouble — instability grows fast (Section 5) while real-world curves are usually gentle. When more bend is genuinely needed, the professional route is a spline or the next chapter's kernel SVR, which bend locally instead of globally — the same flexibility with far less volatility.

8. Strengths, Limits & the Road Ahead

Strengths	Limitations
Keeps all of OLS — fast, exact, interpretable machinery	Global curves: one term must bend everywhere at once
One hyperparameter (degree) — the cleanest complexity dial in ML	Coefficients become unstable and uninterpretable at higher degrees
Captures curvature with minimal code change	Extrapolation is meaningless — x^d owns everything past the range
The perfect vehicle for bias-variance — visible, countable parameters	Powers of x are collinear by construction (tolerated, but watched)

The deepest takeaway is structural: polynomial regression did not add a new algorithm — it added features. Feature engineering as model flexibility is a theme the course will replay at larger scale: Chapter 14's kernel trick computes (in effect) infinite-degree polynomial features without ever building the columns; Chapter 26's neural networks learn their own nonlinear features from data. The bias-variance dial you turned here is the same dial those chapters turn — you now own the vocabulary for reading every one of them.

🛠 Hands-On: Truth or Bluff, All DegreesBeginner

Negotiation reconvened. The candidate now also claims a previous role at "level 12" paying ₹2,500,000. Fit degrees 1–4 (and 9 for the pathology), build the full evidence table, and deliver a verdict on both claims.

Steps

Load Position_Salaries.csv; take Level as X (why skip column 0?), Salary as y.
Fit degrees 1, 2, 3, 4 and 9; tabulate R², prediction at 6.5, prediction at 12.
Print the degree-4 coefficients. Which overfitting signature do they show?
Verdict on claim 1 (level 6.5, ₹160,000): which model(s) support it?
Verdict on claim 2 (level 12, ₹2.5M): what does any polynomial say, and why is the question itself flawed?
Recommend a final degree with a one-paragraph justification citing bias, variance and extrapolation.

View Solution / Walkthrough

import numpy as np, pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

ds = pd.read_csv('Position_Salaries.csv')
X = ds.iloc[:, 1:-1].values.astype(float)   # skip Position: job titles aren't numeric
y = ds.iloc[:, -1].values.astype(float)

for d in [1, 2, 3, 4, 9]:
    p = PolynomialFeatures(degree=d, include_bias=False)
    m = LinearRegression().fit(p.fit_transform(X), y)
    at65  = m.predict(p.transform([[6.5]]))[0]
    at12  = m.predict(p.transform([[12.0]]))[0]
    print(f'deg {d} | R2 {m.score(p.fit_transform(X), y):.4f} '
          f'| @6.5 {at65:,.0f} | @12 {at12:,.0f}')

# Degree-4 coefficients — check the instability signature
p4 = PolynomialFeatures(degree=4, include_bias=False)
m4 = LinearRegression().fit(p4.fit_transform(X), y)
print('deg4: b0=%.0f  coefs=%s' % (m4.intercept_, m4.coef_.round(0)))

Expected results (real):

Table: R² 0.669 / 0.916 / 0.981 / 0.997 / 0.9998; @6.5: 330,379 / 189,498 / 133,259 / 158,862 / 176,785; @12: 775,212 / 1,435,894 / 2,175,030 / 3,035,985 / 3,521,725.
Coefficients (deg 4): 184,167 − 211,002x + 94,765x² − 15,463x³ + 890x⁴ — cancelling waves: signature 2, instability.
Claim 1 (₹160k at 6.5): degrees 3–4 land at ₹133k–159k — plausible; degree 1's ₹330k is the straight line's artefact. Degree 4 (the source's choice) essentially confirms the candidate.
Claim 2 (₹2.5M at 12): degree 3 says 2.18M, degree 4 says 3.04M — the models straddle the claim, which shows how meaningless the question is: every answer is pure x⁴ extrapolation. Verdict: unanswerable from this data; demand comparables, not curves.
Recommendation: degree 4 for interpolation within levels 1–10 (best fit without degree-5's extra volatility), degree ≤ 4 hard-bounded, and an explicit "no predictions past level 10" policy.

🎯 Key Takeaways

"Linear" regression is linear in coefficients, not in x — powers of x are just more features, and OLS fits them unchanged.

Degree is a parameter budget: a line through three points is a coincidence; a parabola is a guarantee. More degree = more bending.

Expected test error = Bias² + Variance + irreducible noise. Underfitting is bias; overfitting is variance; the noise nobody removes.

Training error always falls with complexity; test error is U-shaped. The experiment proved it: minimum at the true degree, 180× worse by degree 9.

Overfitting's signatures: near-perfect in-sample fit, cancelling-wave coefficients, and detonating extrapolation — audit all three.

Choose degrees with domain sense and held-out evidence — never in-sample R², which mechanically rewards complexity.