Part III — Classification

Chapter 15 · Naive Bayes

Streets, votes and lifted geometry all decide — this chapter believes. Naive Bayes turns classification into probability: start from how common each class is, weigh the evidence feature by feature, pick the most probable class. Bayes' theorem built by hand (with the medical-test answer that shocks every beginner), the audacious independence assumption that earns the word "naive", and a complete posterior computed by hand from this dataset's real statistics.

📖 Chapter 15 of 33 ⏱ ~90 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Derive and apply Bayes' theorem by hand — including the base-rate surprise in the medical-test example
Define prior, likelihood and posterior, and identify each in a working classifier
State the conditional independence assumption, explain why it is "naive", and why the classifier works anyway
Compute a Gaussian Naive Bayes posterior by hand from real class statistics on this dataset
Explain Laplace smoothing and match the Gaussian, Bernoulli and Multinomial variants to their data types
Read NB's real results — including a scale-invariance fact and the most confident boundary-customer verdict yet

1. From Geometry to Belief

Four classifiers in, and every one has dodged the same question. Logistic regression gave probabilities, but as a means to draw a line; K-NN votes; SVMs measure distances to streets. None of them answers the question a decision maker actually asks: "given everything we know about this customer, what is the probability they buy?" Naive Bayes answers it head-on — class membership as a belief, updated by evidence, in the framework probability was invented for.

The setup is Chapter 2's language made operational. Before looking at any customer, you hold a prior — the base rate of buying (on our training data: 111 of 300 = 37%). Each feature then contributes evidence — how compatible is this age, this salary, with buyers versus non-buyers? Bayes' theorem combines prior and evidence into the posterior: the after-the-evidence probability of each class. The classifier picks the larger posterior. Same data, same 100 test rows — an entirely different way of thinking.

📝

Note — the two-revolution pedigree: the theorem is Thomas Bayes (1763, published posthumously); the "naive" independence assumption and the classifier are roughly 1950s–1960s. It remains, after sixty years, the backbone of spam filters — and Part VII's text chapter will reuse this exact machinery word by word.

2. Bayes' Theorem, Built by Hand

The theorem, in classifier dress:

P(class | features) = P(class) · P(features | class) / P(features)

P(class) — the prior: how common the class is
P(features | class) — the likelihood: what the features look like inside each class
P(features) — the evidence: a normalising constant, identical for every class — so for choosing the winner, posterior ∝ prior × likelihood is all you need

2.1 The medical test — intuition's defeat, by arithmetic

The classic demonstration, fully by hand. A disease affects 1% of the population. A test catches 99% of the diseased (sensitivity) and falsely flags 5% of the healthy (false-positive rate). Your test comes back positive. What is the probability you are diseased? Intuition shouts "about 99%". Bayes says otherwise — and natural frequencies make the arithmetic unarguable. Take 10,000 people:

Figure 15.1 — The natural-frequency tree: the evidence (a positive test) must fight the prior (almost everyone is healthy).

And the same conclusion through the formula, term by term:

P(D | +) = 0.99 × 0.01 / ( 0.99 × 0.01 + 0.05 × 0.99 ) = 0.0099 / 0.0594 = 1/6 ≈ 0.167

⚠

Warning — the base-rate lesson generalises far beyond medicine: fraud detection (0.1% fraudulent transactions), rare-disease screening, defect inspection. Whenever the target class is rare, a "99% accurate" model can be mostly wrong when it flags — because the prior, not the likelihood, dominates. Chapter 18's precision/recall exists precisely to keep this honest; Chapter 2 warned you the vocabulary would return.

3. The Naive Simplification

To apply Bayes to a customer we need the likelihood P(age and salary | class) — a joint distribution over the feature combination. That is the hard part: joint distributions over many features are enormous (curse of dimensionality, again) and estimating them needs oceans of data. The naive move:

P(x₁, x₂ | class) ≈ P(x₁ | class) · P(x₂ | class)

Conditional independence: given the class, each feature is assumed generated independently of the others. Age and salary are, in reality, correlated among buyers — the assumption is false, hence naive. And the classifier has thrived on it for sixty years.

Why does a provably false assumption produce a working classifier? Two reasons, both practical:

Classification needs the argmax, not the probability. To pick the winner we only need to know which product — prior × likelihoods — is bigger. Systematic distortions that inflate both classes' scores cancel in the comparison.
Correlations often distort both classes similarly. If age–salary correlation double-counts evidence for buyers, it roughly double-counts for non-buyers too. The direction of the decision survives even when the magnitudes are fiction.

💡

Pro Tip — conditional ≠ marginal: the assumption is independence given the class, a weaker claim than plain independence. Age and salary can be strongly related overall yet roughly unrelated once you condition on "is a buyer" — the class itself may explain the correlation. Naive Bayes gets credit for less naivety than it's accused of.

4. Gaussian Naive Bayes, by Hand

For continuous features, each per-class likelihood needs a shape. Gaussian NB fits — you guessed it — a Gaussian (normal curve) to every feature within each class, estimated straight from the training rows. The real statistics of our 300 training customers:

Class	n (prior)	Age: mean ± SD	Salary: mean ± SD (₹)
Non-buyers	189 (0.630)	33.5 ± 7.8	60,180 ± 24,558
Buyers	111 (0.370)	46.0 ± 8.6	85,595 ± 42,207

Figure 15.2 — The two class-conditional Gaussians for age (real parameters), with our anchor customer marked at 30.

Now the full posterior for customer (30, ₹87,000), every number real. The Gaussian density, per class, per feature:

L(x | μ, σ²) = (1/√(2πσ²)) · exp( −(x − μ)² / (2σ²) )

	Prior	L(age = 30)	L(salary = 87k)	Score = prior × likelihoods
Non-buyer	0.630	0.0461	8.95 × 10⁻⁶	2.60 × 10⁻⁷
Buyer	0.370	0.0081	9.45 × 10⁻⁶	0.28 × 10⁻⁷

Read the evidence like a trial transcript:

Age is the star witness: density 0.0461 vs 0.0081 — a 30-year-old is 5.7× more likely under the non-buyer curve (Figure 15.2 makes it visual).
Salary is nearly a hung jury: 8.95 vs 9.45 ×10⁻⁶ — ₹87k is slightly more compatible with buyers (1.06×) but the wider buyer salary spread (SD 42k vs 24.5k) flattens their curve, muting the evidence.
The prior tilts no: 0.630 vs 0.370.
Verdict: 2.60 vs 0.28 ×10⁻⁷ — a 9.2 : 1 ratio, so P(no) ≈ 2.60/(2.60+0.28) ≈ 0.90. sklearn's answer: 0.899. Hand and machine agree.

📝

Note — tiny densities are healthy: salary densities look microscopic (10⁻⁶ per rupee) because a continuous density spreads probability over units — this is Chapter 2's normal curve doing real work. Only the ratio between the classes matters; the absolute size cancels in the comparison (and the normaliser P(features) tidies the rest).

5. Laplace Smoothing & the Variants

One landmine remains in the product of likelihoods: a zero. If a feature value (a word, a category) never appeared in a class's training rows, its likelihood is 0 — and one zero annihilates the entire product, regardless of every other feature's evidence. A single absent word would condemn an otherwise obvious spam email. The classic cure, Laplace (add-1) smoothing:

P̂(word | class) = (count + 1) / (total + K)

count = occurrences of the word in the class's rows; total = all words in that class; K = vocabulary size. No probability is ever zero — unseen words get a small, honest share of belief instead of a veto.

Gaussian NB sidesteps zeros by design (a Gaussian density is positive everywhere), which is why this chapter's code needs no smoothing. The variants menu, matched to feature type:

Variant	Feature type	Per-feature model	Home turf
Gaussian (this chapter)	Continuous	Per-class normal curve	Sensor-like measurements, our ages/salaries
Bernoulli	Binary presence/absence	P(present \| class) per feature	Text as word-occurrence flags (Ch 25)
Multinomial	Counts	Per-class word frequencies	Word counts, documents (Ch 25's main event)

🌎

Real World: The canonical spam filter — Multinomial NB over word counts with Laplace smoothing — has been filtering email since the 1990s and still ships in production stacks. Text is NB's perfect habitat: thousands of rare features, and the naive product of small per-word likelihoods scales where joint distributions never could.

📋 Stable content — Reviewed: August 2026

6. Implementation in Python

import numpy as np
import pandas as pd

dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.naive_bayes import GaussianNB
classifier = GaussianNB()
classifier.fit(X_train, y_train)        # one pass: count means/variances/priors. Done.

from sklearn.metrics import confusion_matrix, accuracy_score
y_pred = classifier.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)                               # [[65  3]
                                        #  [ 7 25]]
print(accuracy_score(y_test, y_pred))   # 0.9

# Real posteriors — NB's native language
print(classifier.predict_proba(
    sc.transform([[30, 150000]])))      # [[0.061, 0.939]] — buyer, confidently

# The model's own statistics (Section 4's table, from the object)
print(classifier.theta_)                # per-class, per-feature means
print(classifier.var_)                  # per-class, per-feature variances
print(classifier.class_prior_)          # [0.63, 0.37]

Listing 15.1 — The complete Gaussian NB pipeline (source: naive_bayes.py).

💡

Pro Tip — the scaling surprise: we ran this pipeline without the scaler too: accuracy identical (0.90). Gaussian NB is scale-invariant — standardising is a linear transform, and each class's mean and variance simply absorb it; the likelihood ratio between classes is untouched. The source's scaling does no harm; Chapter 3's table now has its fourth data point: OLS indifferent, trees immune, kernels desperate, Gaussian NB indifferent.

7. Reading the Results

The five-model scoreboard, all real, same split and scaling:

	Logistic (11)	K-NN (12)	SVM lin (13)	SVM RBF (14)	NB (15)
Confusion matrix	[[65,3],[8,24]]	[[64,4],[3,29]]	[[66,2],[8,24]]	[[64,4],[3,29]]	[[65,3],[7,25]]
Accuracy	0.89	0.93	0.90	0.93	0.90
Model essence	Straight line	Local votes	Widest street	Curved street	Per-class Gaussians × priors
Output	Probability	Vote share	Distance	Distance	Posterior probability
Training cost	Iterative (fast)	None (store)	Quadratic-ish	Quadratic-ish	One pass

NB's 0.90 with 7 missed buyers lands with the linear family — no accident: a product of per-feature Gaussians makes the log-posterior quadratic, so NB's boundary is a gentle parabola-shaped curve, one step above straight lines and below RBF's full flexibility. The interesting column is the last two: NB trains in one arithmetic pass (count means, variances, priors — no iterations, no optimisation) yet matches the optimised linear SVM. When data is streaming or huge, that economy is a superpower.

7.1 The boundary customer, fifth opinion

(30, ₹150k) again. NB's posterior: [0.061, 0.939] — the firmest "buyer" verdict any model has given (logistic hedged at 0.491, the linear SVM crept over at +0.21, RBF committed at +1.13, K-NN voted). The mechanism explains the confidence: salary ₹150k sits in the far tail of the non-buyer Gaussian (mean 60k, SD 24.5k — that's 3.7 SDs out) but only 1.5 SDs above the buyer mean (85.6k, SD 42k). One feature's overwhelming evidence, multiplied through the naive product, sweeps the age objection aside. Whether such confidence is calibrated — 93.9% really meaning 93.9% — depends on the independence assumption actually holding, which it doesn't here; NB's directions are trustworthy, its magnitudes optimistic.

⚠

Warning — probabilities with an asterisk: when features are correlated (age–salary correlation here), the naive product double-counts shared evidence, pushing posteriors toward 0 and 1 — over-confident but usually right-facing. If your business acts on the number ("offer credit when P ≥ 0.8"), calibrate first (Chapter 18's territory); if it acts on the ranking, NB's raw verdicts are fine as-is.

8. Strengths, Limits & the Road Ahead

Strengths	Limitations
Trains in one pass — fastest in the course; streaming-friendly	Independence assumption false for correlated features — over-confident posteriors
Native, interpretable probabilities (prior × evidence)	Per-feature Gaussians miss interactions (the joint shape between features)
Scales to thousands of features — the text champion	Zero-frequency trap for categorical data without smoothing
No hyperparameters to speak of; hard to misuse	Boundary flexibility capped near quadratic — no RBF-level curvature

With belief covered, Part III's toolkit now spans lines, neighbourhoods, streets (straight and curved) and probabilities. The next two chapters return to rules — Chapter 8's flowchart, dressed for classification: a decision tree that splits customers with the same variance-reduction greed you already know, then Chapter 17's forest of them. You have met every ingredient already; only the labels have changed from numbers to classes.

🛠 Hands-On: The Belief MachineIntermediate

Compute one complete Naive Bayes verdict by hand — all the way from priors to posterior — then let sklearn referee, and run the medical-test arithmetic that convinces sceptics.

Steps

Extract priors and class-conditional means/SDs from the training set (they're one groupby away). Compare with Listing 15.1's theta_, var_, class_prior_.
By hand: compute the four Gaussian densities for customer (45, ₹87k) under both classes; multiply by priors; obtain the posterior.
Verify against predict_proba for the same customer — hand and machine should land within a rounding whisker.
Explain, in one sentence each, why (30, ₹87k) → NO and (30, ₹150k) → YES despite the same age.
Run the scaled and unscaled pipelines; confirm identical accuracy and explain the invariance.
Medical drill: with prevalence 0.5%, sensitivity 99%, FPR 5%, compute P(D|+) by both the formula and a 100,000-person natural-frequency tree.

View Solution / Walkthrough

import numpy as np, pandas as pd, math
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

ds = pd.read_csv('Social_Network_Ads.csv')
X, y = ds.iloc[:, :-1].values, ds.iloc[:, -1].values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)

# Step 1: priors and per-class stats
for cls in [0, 1]:
    m = X_tr[y_tr == cls]
    print(cls, 'n=%d  age %.1f±%.1f  salary %.0f±%.0f' %
          (len(m), m[:,0].mean(), m[:,0].std(),
           m[:,1].mean(), m[:,1].std()))

# Step 2: hand posterior for (45, 87000)
def gauss(x, mu, sd):
    return math.exp(-(x-mu)**2/(2*sd**2)) / (sd*math.sqrt(2*math.pi))
scores = {}
for cls, prior in [(0, 189/300), (1, 111/300)]:
    m = X_tr[y_tr == cls]
    s = prior * gauss(45, m[:,0].mean(), m[:,0].std()) \
              * gauss(87000, m[:,1].mean(), m[:,1].std())
    scores[cls] = s
    print('class', cls, 'score %.3e' % s)
print('P(buy) = %.4f' % (scores[1]/(scores[0]+scores[1])))

# Step 3: referee
nb = GaussianNB().fit(X_tr, y_tr)
print(nb.predict_proba([[45, 87000]]))

# Step 6: medical drill
prev, sens, fpr = 0.005, 0.99, 0.05
pd_pos = sens*prev / (sens*prev + fpr*(1-prev))
print('P(D|+) = %.4f  (tree: 4950 sick->495 +; 99500 well->4975 +; 495/5470)' % pd_pos)

Expected results (real):

Step 1: priors 0.630/0.370; non-buyers age 33.5±7.8, salary 60,180±24,558; buyers 46.0±8.6, 85,595±42,207 — matching theta_/var_ exactly.
Steps 2–3: hand computation gives P(buy) ≈ 0.62 — a hesitant yes; sklearn reports [0.381, 0.619]. Agreement within rounding.
Step 4: (30, 87k): age evidence 5.7× for no, salary nearly neutral, prior for no → NO. (30, 150k): the same age penalty, but salary sits 3.7 SD above the non-buyer mean vs 1.5 SD above the buyer mean — the tail evidence overwhelms → YES (0.939).
Step 5: both pipelines 0.90 — the linear standardisation is absorbed by each class's μ and σ; likelihood ratios unchanged.
Step 6: P(D|+) = 495/5470 ≈ 0.0906 — halving the prevalence halved the answer again; rarer disease, emptier positives. The base-rate lesson compounds.

🎯 Key Takeaways

Bayes: posterior ∝ prior × likelihood. The 99%-sensitive test on a 1% disease yields P = 1/6 — base rates dominate rare-event detection.

"Naive" = conditional independence of features given the class. False in practice, survivable because classification needs only the argmax.

Gaussian NB fits a normal curve per feature per class (real: ages 33.5 vs 46.0) and multiplies: our hand posterior 0.90 met sklearn's 0.899.

Laplace smoothing ((count+1)/(n+K)) stops one unseen word from vetoing the product — essential for categorical/text NB, unneeded for Gaussians.

Gaussian NB is scale-invariant (real: 0.90 with or without the scaler) and trains in one pass — the fastest learner in the course.

Its posteriors are directionally right but over-confident under correlated features — trust the ranking, calibrate the number (Ch 18).