Chapter 5 · Multiple Linear Regression

1. From One Feature to Many

Chapter 4 predicted salary from one number. The 50_Startups problem is richer: predict annual Profit from R&D Spend, Administration Spend, Marketing Spend, and State (New York / California / Florida) — 50 startups, four features, one continuous target. Multiple linear regression (MLR) is the direct extension: a linear coefficient for every feature, all fitted by the same least-squares logic.

1.1 The phrase that unlocks every coefficient

A slope in MLR is no longer "the average change in y per unit of x" full stop — other features could be moving too. The correct reading adds four words: holding the other features constant. Each coefficient isolates its feature's contribution ceteris paribus. This is precisely what makes MLR more valuable than running five separate simple regressions: it disentangles overlapping influences.

2. Categorical Predictors & the Dummy Trap — Solved

Chapter 3 one-hot encoded State into three dummies and promised Chapter 5 would spring the trap. Here it is. With all three columns present, every row sums the dummies to exactly 1 — which is exactly what the intercept's constant column does:

The design matrix loses rank: the "equation" XᵀX b = Xᵀy no longer has a unique solution — infinitely many coefficient sets fit identically, because any amount can be shuffled between the intercept and the three dummies without changing a single prediction. This is the dummy variable trap: a special case of multicollinearity created by encoding, not by the world.

2.1 The cure: drop one category

import pandas as pd

dummies = pd.get_dummies(df['State'], drop_first=True)   # drops 'California'
# -> columns: Florida, New York.  California is implied by both being 0.

Listing 5.1 — Two dummies carry all three categories' information.

The dropped category becomes the baseline (reference) category, and every dummy coefficient changes meaning — it is now a difference against that baseline:

• Intercept b₀: predicted Profit for the baseline state (California) when spends are 0
• Florida coefficient: Profit difference between Florida and California, holding spends constant
• New York coefficient: same, New York vs California

3. New Assumptions for Multiple Features

Everything from Chapter 4's LINE suite carries over unchanged — linearity now means "linear in each coefficient", independence, normality and equal variance still apply to the residuals, and the residual-vs-fitted plot remains the diagnostic workhorse. Multiple features add three new obligations:

The sample-size rule deserves a sentence of arithmetic: 50 rows ÷ 3 numeric predictors ≈ 17 rows per predictor — acceptable; add three dummies and the ratio falls to 10 — the boundary. Small datasets make every added feature expensive, which is exactly why Section 6's elimination matters here.

4. Multicollinearity & the VIF

Multicollinearity is a relationship among predictors — the target is not involved. Pairwise correlations reveal the obvious cases (R&D ↔ Marketing at 0.724), but collinearity can hide in combinations: three features where any pair looks innocent, yet one is nearly a linear mix of the other two. The Variance Inflation Factor (VIF) catches what pairwise correlation misses, per feature:

The conventional thresholds: VIF > 5 — investigate; VIF > 10 — serious, act. Despite the eye-catching 0.724 pairwise correlation, all three features pass comfortably. The lesson generalises: correlation that looks large in a scatter can still leave a feature plenty of independent variation — let the VIF, not the anxiety, decide.

from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

X = sm.add_constant(startups[['R&D Spend', 'Administration', 'Marketing Spend']])
for i, col in enumerate(X.columns):
    print(col, round(variance_inflation_factor(X.values, i), 2))

Listing 5.2 — Computing VIFs with statsmodels (always add the constant column first).

5. Adjusted R² & the p-Value Toolkit

5.1 Adjusted R² — the penalty clause

Chapter 4's Trap 2 returns with teeth: training R² never falls when you add a predictor — even pure noise. With five features and 50 rows, R² alone cannot distinguish signal from arithmetic. Adjusted R² amends the contract:

5.2 The statsmodels summary — a guided tour

scikit-learn answers "what is the prediction?" statsmodels answers "what do we know about the coefficients?" Here is the real summary for the full model (three spends + Florida and New York dummies, California baseline), annotated column by column:

• coef — bⱼ: the ceteris-paribus effect (Section 1's phrase).
• std err — the coefficient's own uncertainty: how much bⱼ would wobble across fresh samples.
• t = coef / std err — Chapter 2's t-statistic reborn: signal over noise. R&D's 17.4 is a shout; Administration's −0.5 is a whisper.
• P>|t| — the two-tailed p-value for H₀: βⱼ = 0. R&D's 0.000: overwhelmingly nonzero. Administration's 0.608: indistinguishable from zero.
• [0.025, 0.975] — the 95% confidence interval for βⱼ; notice Administration's straddles zero, matching its p-value (Chapter 2's two views of one uncertainty).

The summary's bottom rows carry model-wide verdicts:

• F-statistic = 169.9 (p ≈ 1.3×10⁻²⁷) — a joint test of H₀ "all βⱼ = 0 simultaneously": the model as a whole is overwhelmingly better than predicting the mean.
• R² = 0.9508, Adj R² = 0.9452 — the ledger and the penalty.
• AIC = 1062.8 — an information criterion: fit minus a complexity charge, comparable across models on the same data — lower is better. Section 6 will use it.

6. Feature Selection: Backward Elimination

The full model spends five predictors, yet its summary says three of them look like noise. Backward elimination formalises the pruning, p-value by p-value:

1. Choose a significance level for staying (conventionally SL = 0.05)
2. Fit the full model with all predictors
3. Delete the predictor with the highest p-value, if it exceeds SL
4. Refit and repeat step 3
5. Stop when every remaining p-value ≤ SL

Run on 50_Startups (real output, full dataset, California baseline):

6.1 The borderline: p = 0.060

Here is the honest fork. At a strict SL = 0.05, Marketing goes and the final model is R&D alone. At SL = 0.10 — a threshold many practitioners deliberately choose for retention ("be a little generous before firing a feature") — elimination stops at R&D + Marketing. The tiebreakers say keep it:

The two-feature model wins adjusted R², wins AIC, and wins the held-out test set — the source model's extra three predictors bought in-sample glitter and paid for it in generalisation. Parsimony is not asceticism; it is arithmetic.

8. Reading, Reporting & What's Next

With the chosen final model (R&D + Marketing, full-data fit), the professional report writes itself:

Notice what got reported that tutorials omit: the null findings (Administration, State) and the borderline call (Marketing at p = 0.06, kept on adjusted R², AIC and test evidence). Professionals are paid for the judgement layer, not the .fit() call.

The road ahead

MLR still draws straight planes. When the shape of the relationship bends — and Chapter 6's salary-vs-level data bends hard — polynomial regression keeps the linear machinery and bends the feature space, introducing the bias-variance tradeoff that governs everything from Chapter 6 to Chapter 33. Before that: this chapter's quiz, thirty questions, every section fair game.