Chapter 4 · Simple Linear Regression

1. Regression: Predicting Numbers

Regression is supervised learning with a continuous target: predicting a salary, a price, a temperature, a demand level. The complement — predicting a category — is classification, and it takes over in Chapter 11. Simple linear regression (SLR) is regression in its purest form: exactly one numeric feature, and the assumption that the relationship with the target is a straight line.

That austerity is deliberate. Because SLR has only two parameters, every idea in this chapter — estimation, assumptions, fit, diagnostics — can be computed and seen. Multiple regression (Chapter 5) adds features, polynomial regression (Chapter 6) bends the line, but the vocabulary and the validity questions are all invented here.

1.1 The running problem: salary from experience

Salary_Data.csv holds 30 employees: years of experience (1.1 to 10.5) and annual salary (₹37,731 to ₹122,391). The business question: given a candidate's years of experience, what salary should we expect?

2. The Model & the Least-Squares Line

The SLR model says: the mean of y at any x lies on a straight line.

Real points do not sit on the line. The vertical gap between an observed value and its prediction is the residual:

2.1 What "best fit" means — ordinary least squares

Infinitely many lines pass near these points. Ordinary least squares (OLS) picks the one minimising the sum of squared residuals:

Why squared? Squaring makes every residual positive (so misses in either direction count against the fit), penalises large misses disproportionately — one error of 4 costs more than two errors of 2 — and, crucially, produces a smooth, differentiable objective whose minimum can be found exactly, with calculus taught in any first course:

1. Take the partial derivative of Q with respect to b₀ and set it to zero
2. Take the partial derivative with respect to b₁ and set it to zero
3. Solve the two equations — the "normal equations" — for the two unknowns

The solution — the least-squares estimates — is remarkably compact:

3. Worked by Hand: Five Employees

Before Python runs anything, derive the line yourself once. Five employees, years of experience x and salary y (in ₹ lakh per annum — small numbers by design):

Step 1 — slope:

Step 2 — intercept (line must pass through (x̄, ȳ) = (3, 7)):

Step 3 — predictions and residuals:

Step 4 — R² (derived properly in Section 5):

You have just fitted a regression with pencil and paper: every additional year of experience is associated with +₹1.7 lakh of salary on average; the model explains 96.3% of the salary variation across these five employees. Nothing that follows — not sklearn, not a million rows — changes the idea, only the arithmetic.

4. The Assumptions — LINE

OLS always returns a line — on any data, no questions asked. The assumptions decide whether that line means anything. The classic mnemonic LINE organises the four conditions that make OLS estimates trustworthy, predictions valid, and inference (p-values, Chapter 5) honest:

The workhorse check is a single picture: residuals on the y-axis against fitted values on the x-axis. A healthy model shows a formless horizontal band around zero. Every systematic shape is the residuals trying to tell you which assumption is bleeding:

5. R² — the Variance Ledger

How good is the line? The elegant answer decomposes the target's total variation into explained and unexplained parts. Define the total sum of squares and its two children:

The coefficient of determination is the explained share:

5.1 What counts as a good R²?

The honest answer is domain-relative: a 0.4 can be a publishing event in one field and a rounding error in another. Judge R² against what is explainable in principle, not against 1.0.

5.2 The three R² traps

• Trap 1 — high R², invalid model. Anscombe's quartet (Chapter 2): a curved dataset can carry the same R² as a clean linear one. R² measures fit, not correctness. The residual plot is the arbiter.
• Trap 2 — in-sample R² only rises. Add any feature — even random noise — and training R² can only increase or stay flat, because OLS can always ignore it at worst... and usually finds some accidental correlation. Which is why we evaluate on the test set, and why Chapter 5 introduces adjusted R², which penalises predictor count.
• Trap 3 — R² says nothing about scale. R² = 0.96 on salaries could still mean ±₹20,000 typical misses. Chapter 10's metrics (MAE, RMSE) speak in target units; R² speaks in variance shares. Report both.

7. Reading & Reporting the Model

7.1 Translating the coefficients

With b₀ = 26,816 and b₁ = 9,346:

• Slope: "Each additional year of experience is associated with about ₹9,346 higher salary, on average, across the observed range." — this is the sentence the business asked for.
• Intercept: "A employee with zero years of experience would earn ₹26,816." — grammatically fine, substantively suspect: no one in the data has less than 1.1 years, so b₀ is an extrapolation to a region the model never saw. The line must pass through (x̄, ȳ); the intercept is where that geometric fact happens to meet x = 0. Interpret it only when x = 0 is meaningful and observed.

7.2 The diagnostic pass — assumptions on real data

residuals = y_train - regressor.predict(X_train)

# L & E: residuals vs fitted — want a flat, formless band
plt.scatter(regressor.predict(X_train), residuals)
plt.axhline(0, color='grey', linestyle='--')
plt.xlabel('Fitted salary'); plt.ylabel('Residual')
plt.title('Residuals vs Fitted'); plt.show()

# N: Q-Q plot — want points hugging the diagonal
from scipy import stats
stats.probplot(residuals, plot=plt)
plt.title('Q-Q plot of residuals'); plt.show()

Listing 4.4 — The two-plot diagnostic pass from Section 4, on the training residuals.

On this dataset the band stays reasonably flat with mild widening at the right (senior salaries spread more — Section 4's Real World box, materialised), and the Q–Q points track the diagonal with light tail wobble. Verdict: assumption concerns are minor here, log-salary would be the professional refinement, and the honest report says so.

7.3 The one-paragraph report

8. Strengths, Limitations & the Road Ahead

Keep SLR as your permanent first model — the two-minute baseline that contextualises everything later. The next three chapters grow it: Chapter 5 adds features (and the multicollinearity, dummy-trap and feature-selection questions that come with them), Chapter 6 bends the line and formally introduces the bias-variance tradeoff, and Chapters 7–9 swap the line for more flexible learners — each judged against this chapter's transparent yardstick.