Chapter 2 · Statistics Primer for Machine Learning

1. Populations, Samples & Generalisation

In statistics, a population is the complete set of entities we want to draw a conclusion about — every customer of a retailer, every transaction ever made, every patient with a condition. A sample is the subset of that population we actually observe. Almost always, the population is too large (or too expensive, or does not exist yet — think future customers) to measure entirely, so we reason from the sample to the population.

This is exactly what a machine learning model does. A training set is a sample; the model's learned parameters are estimates of some underlying truth; and the promise that a model scoring well on its sample will also score well on new, unseen cases is the statistical promise of generalisation. When a model fails to generalise (Chapter 1's overfitting), that is a failure of statistical reasoning, not a programming bug.

Parameters vs statistics

Numbers that describe a population are called parameters; the same quantities computed from a sample are called statistics. Different symbols keep the distinction visible:

2. Data Types & Measurement Scales

Before any statistic or model is computed, you must know what kind of numbers you are holding. The classic hierarchy of measurement scales has four levels, each supporting strictly more operations than the last:

The temperature example makes the interval/ratio boundary concrete: 20 °C is not "twice as hot" as 10 °C, because 0 °C is an arbitrary anchor. Salary, by contrast, has a true zero — ₹40,000 really is twice ₹20,000 — so it sits on a ratio scale and ratios are meaningful.

Why a data-type column matters in machine learning

Data type decides encoding, and encoding decides model behaviour. Label encoding a nominal variable (France=0, Germany=1, Spain=2) injects a fake order and a fake metric — "Spain is more than Germany" — that linear and distance-based models will take literally. One-hot encoding avoids that at the cost of extra columns. Chapter 3 turns this into a complete decision guide; here, just internalise the principle:

One more distinction, orthogonal to scale: continuous variables take any value in a range (salary, age), while discrete variables take separated values (number of children, count of purchases). Regression targets are typically continuous; classification targets are discrete — the split the next two Parts of this course are organised around.

3. Descriptive Statistics: Centre & Spread

Descriptive statistics compress a variable into a few interpretable numbers. The two questions they answer are "where is the middle?" (centre) and "how scattered are the values?" (spread).

3.1 Measures of centre

The mean x̄ is the arithmetic average — the balance point of the data. The median is the middle value when the data are sorted — half the values lie below it. The mode is the most frequent value, the only centre measure defined for nominal data.

The mean uses every value — which is both its power and its weakness. A single extreme value drags it; the median resists. House prices, incomes and delivery times are classic right-skewed variables where the mean sits noticeably above the median (Section 4.4). Quick rule:

3.2 Measures of spread

The range (max − min) is the crudest spread measure — it depends entirely on the two extremes. The interquartile range (IQR) is the width of the middle 50% of the data: Q3 − Q1, robust to outliers. The variance and standard deviation (SD) measure average squared distance from the mean:

Variance is in squared units (rupees², years²) — fine for mathematics, poor for communication. The SD undoes the squaring: "salary has a mean of ₹76,000 with SD ₹14,000" is a sentence a manager can use. In ML, squared units return as the loss — mean squared error in Chapter 10 — and variance is the raw material of PCA in Chapter 28.

3.3 Worked example — by hand

Eight years-of-experience values from a salary dataset (in years):

Centre. Sorted already: mean = 40 / 8 = 5.0; median = (4 + 5) / 2 = 4.5; mode = 4 (three occurrences). Mean above median is an early hint of right skew.

Spread. Deviations from the mean: −3, −1, −1, −1, 0, 0, 2, 4. Squared: 9, 1, 1, 1, 0, 0, 4, 16 → sum = 32.

(Had this been the entire population, variance would be 32 / 8 = 4 and SD exactly 2 — the n vs n−1 distinction in one calculation.)

Quartiles and outliers. Lower half {2,4,4,4} → Q1 = 4; upper half {5,5,7,9} → Q3 = 6; IQR = 2. The 1.5 × IQR fences sit at 4 − 3 = 1 and 6 + 3 = 9. Every value lies within [1, 9] — no outliers by this rule. Change the 9 into a 90 and the fences would flag it instantly.

3.4 In Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Salary_Data.csv')
years = dataset['YearsExperience']

print(years.describe())          # count, mean, std, min, quartiles, max
print('Median :', years.median())
print('Mode   :', years.mode().tolist())

# Outlier fences via the 1.5 x IQR rule
q1, q3 = years.quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print('Outliers:', years[(years < lower) | (years > upper)].tolist())

Listing 2.1 — Descriptive statistics and the IQR outlier rule with pandas.

4. The Normal Distribution & Standardisation

4.1 Why the bell curve keeps appearing

The normal (Gaussian) distribution is the symmetric, bell-shaped curve defined by two parameters — mean μ (location of the peak) and standard deviation σ (width of the bell). It recurs for a deep reason: the Central Limit Theorem says that averages of many independent contributions trend toward normality regardless of the individual components' shapes. Measurement errors, aggregates of many small factors, and — crucially — the sampling distributions of many statistics (means, regression coefficients) are approximately normal. That is why hypothesis tests in Section 6 lean on it.

4.2 The empirical rule and z-scores

For any normal distribution, the empirical rule (Figure 2.1) fixes how much probability sits within each SD band: 68% within ±1σ, 95% within ±2σ, 99.7% within ±3σ. This turns the SD into a universal ruler: "three standard deviations from the mean" flags the rarest 0.3% of cases.

The z-score formalises that ruler — how many standard deviations an observation sits from its mean:

Quick check: an employee earns ₹85,000 where the mean is ₹70,000 and SD is ₹5,000 → z = (85,000 − 70,000) / 5,000 = +3.0: the top 0.15% of a normal salary distribution. Rare — and immediately quantified.

4.3 Standardisation — the bridge to feature scaling

Compute z-scores for every value of a variable and you have standardised it: the result always has mean 0 and SD 1. Chapter 3 will implement this as StandardScaler and explain when ML algorithms need it (distance- and gradient-based ones) — but the mathematics you will be running is exactly this z-score formula.

import pandas as pd

dataset = pd.read_csv('Salary_Data.csv')

# Manual z-scores (exactly what StandardScaler computes)
z = (dataset['Salary'] - dataset['Salary'].mean()) / dataset['Salary'].std()
print('Mean of z:', round(z.mean(), 6))    # ~0
print('SD of z  :', round(z.std(), 6))     # ~1
print('Most extreme salary, z =', round(z.max(), 2))

Listing 2.2 — Standardising a column by hand; verify mean 0, SD 1.

4.4 Skewness — when the bell is lopsided

Real ML data is often not symmetric. Right (positive) skew — a long tail of high values — pulls the mean above the median; incomes, house prices and claim sizes behave this way. Left (negative) skew does the opposite. Skewed targets harm linear models (their residuals become asymmetric, violating a Chapter 4 assumption), and the standard fixes — log or square-root transformations — compress the long tail.

5. Correlation, Covariance & Causation

5.1 Covariance — direction without scale

Do two variables move together? Covariance averages the product of paired deviations from each mean:

When values above the x-mean tend to pair with values above the y-mean, the products are mostly positive → positive covariance. But covariance inherits its units (years × rupees) and is unbounded, so "is 4,80,000 big?" has no general answer — you cannot compare it across datasets or even across variable pairs.

5.2 Pearson's correlation coefficient — the fix

Divide covariance by both standard deviations and the units cancel, leaving Pearson's r, bounded to [−1, +1]:

5.3 Worked example — r by hand

Five employees, years of experience x and performance score y:

Interpretation: a moderately strong positive linear association. Squaring r gives r² ≈ 0.60 — "about 60% of the variation in performance scores lines up with variation in experience." That r → r² move is exactly how R² (Chapter 4's headline regression metric) is built.

5.4 Correlation is not causation

import pandas as pd

dataset = pd.read_csv('Salary_Data.csv')
print(dataset.corr(numeric_only=True))
#        YearsExperience  Salary
# YearsExperience        1.00    0.98
# Salary                 0.98    1.00

Listing 2.3 — Correlation matrix: experience and salary are strongly associated (r ≈ 0.98) — association, not yet a causal claim.

6. Hypothesis Testing & p-Values

Descriptive statistics summarise what you saw; hypothesis testing asks whether what you saw generalises. The machinery has five parts, and every part maps onto an ML idea later in the course.

6.1 The logic

1. Null hypothesis (H₀) — the skeptic's position: "nothing special is happening" (no difference, no effect, means are equal).
2. Alternative hypothesis (H₁) — what you suspect: there is an effect / a difference.
3. Test statistic — a single number (t, z, χ², F…) measuring how far the sample fell from what H₀ predicted.
4. p-value — if H₀ were true, the probability of a sample at least this extreme.
5. Decision — reject H₀ when p < α (typically 0.05); otherwise fail to reject. We never "accept" H₀ — absence of evidence is not evidence of absence.

6.2 Errors, power and confidence

α and β pull against each other: lowering α from 0.05 to 0.01 makes false positives rarer and false negatives more common. A 95% confidence interval expresses the same uncertainty in estimate space: a procedure that builds 95% CIs will capture the true parameter in 95 of 100 samples. It is not "a 95% chance the true value sits in this one interval" — the parameter is fixed; the interval is the random object.

6.3 Worked example — one-sample t-test

A delivery platform promises 30-minute delivery. You sample n = 25 orders: mean x̄ = 32 min, sample SD s = 5 min. Is the promise broken?

At α = 0.05 with df = 24, the one-tailed critical t is 1.711 and the two-tailed critical t is 2.064. Our t = 2.00 is beyond 1.711 but short of 2.064:

• One-tailed (H₁: μ > 30, the right question here — we only care about slowness): reject H₀. Evidence of slower delivery.
• Two-tailed (H₁: μ ≠ 30): fail to reject. Not enough evidence either way.

Same data, different question, different answer — which is why the hypothesis is chosen before seeing the data, never shopped for afterwards.

import numpy as np
from scipy import stats

times = np.array([31, 29, 35, 28, 33, 30, 36, 27, 34, 31,
                  33, 30, 29, 37, 32, 30, 28, 35, 33, 31,
                  30, 32, 34, 29, 31])          # n = 25

t_stat, p_two = stats.ttest_1samp(times, popmean=30)
print('t =', round(t_stat, 3), ' two-tailed p =', round(p_two, 4))
print('one-tailed p =', round(p_two / 2, 4))   # H1: mean > 30

Listing 2.4 — One-sample t-test with scipy; halve the two-tailed p for the one-sided alternative.

8. From Statistics to Machine Learning

Nothing in this chapter was decoration. Every concept returns with a machine learning job title:

Treat this chapter as a dictionary you will keep consulting. When Chapter 3 asks why scaling matters, come back to z-scores; when Chapter 5 prints p-values beside every coefficient, come back to Section 6; when Chapter 4 reports R², remember it is r² wearing a regression costume.