Part I — Foundations & Data Preprocessing

Chapter 3 · Data Preprocessing in Python

Every model in this course is only as good as the matrix fed into it. This chapter turns raw, messy, mixed-type data into a model-ready numeric matrix — missing values, categorical encoding, train/test splitting and feature scaling — and teaches the one professional habit that separates practitioners from tutorial-followers: fit on the training set, transform the test set.

📖 Chapter 3 of 33 ⏱ ~90 min read Mostly Core Stable 📝 Quiz · 30 MCQs

🎯 Learning Objectives

Split a dataset into the feature matrix X and target vector y, and explain the NumPy convention every sklearn call expects
Classify missing-data mechanisms (MCAR, MAR, MNAR) and choose between deletion and mean/median/mode imputation defensibly
Encode categorical data correctly — one-hot for features, label encoding for the target — and explain the dummy variable trap in one sentence
Split data into training and test sets with the right test size, reproducibility, and (when needed) stratification
Apply standardisation and normalisation, and state which algorithms need scaling and which are naturally immune
Order all preprocessing steps to avoid data leakage — and recognise it in code that looks perfectly fine

1. Why Preprocessing Decides the Project

Models do not "see" your business problem — they see a numeric matrix. Real data arrives with gaps, text labels, wildly different scales and more rows than any single computation should trust. Preprocessing is the discipline of converting raw data into that matrix without distorting or leaking information. Get it wrong and the most sophisticated algorithm will learn your mistakes with perfect confidence.

The five jobs of preprocessing, in the order this chapter teaches them:

Load & shape — feature matrix X, target vector y
Repair — missing values
Translate — categorical text → numbers
Divide — training set vs test set
Rescale — comparable ranges for distance- and gradient-based algorithms

🌎

Real World: Surveys of working data scientists consistently find that data preparation — not modelling — consumes the majority of project time. Every hour invested here repays itself later: a clean matrix makes model behaviour interpretable, while a dirty one makes every downstream metric suspect.

📝

Note: This chapter builds on Chapter 2's vocabulary — measurement scales (which encoding?), z-scores (what standardisation computes), and samples vs populations (why we hold out a test set). If those feel fuzzy, a quick re-read of Chapter 2's Sections 2, 4 and 1 will pay off.

2. Importing Data & the Feature/Target Split

This chapter's running dataset is the classic Data.csv — ten customer records with two gaps deliberately planted:

Row	Country	Age	Salary	Purchased
1	France	44	72,000	No
2	Spain	27	48,000	Yes
3	Germany	30	54,000	No
4	Spain	38	61,000	No
5	Germany	40	missing	Yes
6	France	35	58,000	Yes
7	Spain	missing	52,000	No
8	France	48	79,000	Yes
9	Germany	50	83,000	No
10	France	37	67,000	Yes

The target — the thing we want to predict — is Purchased. Everything else is a feature (predictor, independent variable). Every sklearn workflow starts by separating them:

import numpy as np
import pandas as pd

dataset = pd.read_csv('Data.csv')

X = dataset.iloc[:, :-1].values    # features: all rows, all columns except the last
y = dataset.iloc[:, -1].values     # target: all rows, last column only

Listing 3.1 — The feature matrix X and target vector y.

Reading `iloc` — one line of syntax you will use forever

iloc is integer-location indexing: [rows, columns]. The slice :-1 means "everything up to, but not including, the last element" — so [:, :-1] selects all rows and all columns except the final one, and [:, -1] selects all rows of just the final column. Appending .values extracts the underlying NumPy array from the pandas DataFrame, because scikit-learn's estimators are built around NumPy arrays (modern sklearn also accepts DataFrames, and remembers column names for you — but the array convention this course uses is universal).

The first look — always

print(dataset.info())
print(dataset.isnull().sum())

Listing 3.2 — dtypes, row counts, and a per-column missing-value census.

💡

Pro Tip: dataset.info() is the single most useful line in pandas: row count, column dtypes, and — by comparing "non-null" counts against the total — exactly where the gaps are. Chapter 2's descriptive audit (describe()) completes the portrait. Run both before touching any model.

3. Missing Data: Mechanisms & Imputation

Data.csv has two gaps: one salary (row 5) and one age (row 7). Before choosing a fix, a professional asks why the values are missing — because the mechanism decides the treatment:

Mechanism	Meaning	Example	Safe fix
MCAR Missing Completely At Random	Missingness unrelated to anything, observed or not	A sensor glitch dropped one reading	Deletion or simple imputation is low-risk
MAR Missing At Random	Missingness explained by other observed variables	Younger customers skip the income field more often	Impute using related columns (or model-based imputation)
MNAR Missing Not At Random	Missingness depends on the unobserved value itself	Very high earners refuse to disclose salary	Hardest case — impute and add a missing-indicator column; deletion biases results

3.1 Deletion vs imputation

Drop rows when very few are affected (a working rule: under ~5% of rows) and the mechanism is plausibly MCAR. More than that and you start throwing away signal — and biasing the sample if the missingness is not random.
Drop columns when a feature is mostly missing (roughly half or more) — it cannot be rescued honestly.
Impute otherwise: fill the gap with an informed guess, keeping the row's other information.

⚠

Warning: Never impute the target y. A guessed label is fabricated ground truth — the model will be graded against fiction. Rows with a missing target are dropped, full stop. And never forget imputed values are guesses: heavy mean-imputation shrinks variance and weakens correlations (Chapter 2), making models confidently wrong.

3.2 Which value to impute?

Mean — symmetric, outlier-free numeric columns (the default reflex)
Median — skewed numeric columns or when outliers are present (Chapter 2's lesson: the median resists)
Mode (most_frequent) — categorical columns, where a mean makes no sense
Constant (e.g., 0 or "Unknown") — for MNAR cases, paired with a 0/1 missing-indicator column that preserves the missingness signal

3.3 Worked by hand — Data.csv

Nine observed ages sum to 349, so the mean age is 349 / 9 ≈ 38.78 — the value that will fill row 7. Nine observed salaries sum to 574,000, so the mean salary is 574,000 / 9 ≈ 63,777.78 — filling row 5. With only ten rows we would hesitate to trust these; with thousands, mean-imputation of a handful of MCAR gaps is routine.

3.4 In Python — SimpleImputer and the fit/transform pattern

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer.fit(X[:, 1:3])                       # learn: compute column means from Age & Salary
X[:, 1:3] = imputer.transform(X[:, 1:3])     # apply: replace NaNs with the learned means

Listing 3.3 — Mean-imputation of the numeric columns (source: data_preprocessing_tools.py).

Note the two-step dance — this is your first meeting with sklearn's central pattern, and it is worth slowing down for:

.fit() learns parameters from data (here: one mean per column) and stores them inside the object
.transform() applies the stored parameters to produce new data
.fit_transform() does both in one call — legitimate when fitting and transforming the same data set

The slice X[:, 1:3] targets only columns 1 and 2 (Age, Salary) — the numeric ones. The Country column (index 0) is never touched by the imputer: a mean of "France, Spain, …" is meaningless, which is why categorical gaps need strategy='most_frequent' instead.

4. Encoding Categorical Data

Models compute with numbers, so France and No must become numeric before training. Chapter 2 established the principle — scale dictates encoding — and now it earns its keep, because the feature and the target follow different rules.

4.1 The target: label encoding (safe here)

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
y = le.fit_transform(y)      # 'No' -> 0, 'Yes' -> 1

Listing 3.4 — The binary target becomes 0/1.

Why is plain label encoding fine for the target when Chapter 2 forbade it for features? Because the model never computes distances between target values — it only needs class codes to compare predictions against. Features, by contrast, enter the maths directly, and there the encoding's geometry matters.

4.2 Nominal features: one-hot encoding

Label-encoding Country as France=0, Spain=1, Germany=2 would invent an order ("Germany > Spain") and a distance ("Germany is twice Spain") that exist nowhere in reality — and linear and distance-based models will use them anyway (Chapter 2's warning, now with consequences). One-hot encoding dodges the problem: one binary column per category, exactly one of which "fires":

Country	France	Germany	Spain
France	1	0	0
Germany	0	1	0
Spain	0	0	1

Every pair of countries is now equidistant — no fake order, no fake ratios. The cost: dimensionality grows with the number of categories, and for very high-cardinality columns (thousands of product IDs) one-hot becomes impractical — an advanced problem noted in Chapter 5.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

ct = ColumnTransformer(
    transformers=[('encoder', OneHotEncoder(sparse_output=False), [0])],
    remainder='passthrough')
X = np.array(ct.fit_transform(X))

Listing 3.5 — One-hot the Country column; pass the rest through untouched.

ColumnTransformer applies different transformations to different columns — here, "encode column 0, leave the rest alone" (remainder='passthrough'). Without that argument, the untouched columns would be silently dropped — a classic silent bug. In modern sklearn (≥ 1.2), sparse_output=False asks for a dense array so the result slots straight back into X.

X now has five columns: three country dummies + Age + Salary. And the encoding gracefully handles a category appearing in the test set that was never seen in training — configure handle_unknown='ignore' when that risk exists.

📝

Note — the dummy variable trap: keeping all three country dummies makes them sum to 1 in every row — perfect redundancy (statisticians call it perfect multicollinearity), which destabilises linear models in Chapter 5. The cure is to drop one category (drop='first'): two dummies carry all the information. Chapter 5 demonstrates the trap and the cure properly; here you only need the one-sentence version.

5. Splitting into Training and Test Sets

Chapter 1's promise — models are judged on unseen data — becomes a physical operation here: hide a slice of rows from the model during training, then use it exactly once as a stand-in for the future.

Figure 3.1 — The train/test split: the test set stands in for the population of future data.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,      # 20% of rows held out for testing
    random_state=1)     # seed the shuffler -> reproducible split

Listing 3.6 — The split (source: data_preprocessing_tools.py).

5.1 Choosing test_size and random_state

Dataset size	Typical test_size	Why
Tens of rows (like Data.csv)	0.2 – 0.3	Small data: every training row is precious, but the test estimate is coarse — treat it as indicative (Chapter 31's cross-validation is the real fix)
Thousands to tens of thousands	0.2 – 0.25	The classic default; both sets statistically healthy
Millions of rows	0.005 – 0.02	A 1% test set is already tens of thousands of rows; keep the rest for training

random_state seeds the shuffling so the split is identical on every run — the reproducibility habit this course keeps with every model's random_state. Change the seed and the specific rows swap; on small data the swap can visibly move the score, which is exactly the fragility cross-validation (Chapter 31) addresses.

💡

Pro Tip: For classification, add stratify=y: it preserves the class proportions in both sets, so a 90/10 class split stays 90/10 in train and test. From Chapter 11 onward this is standard practice whenever classes are imbalanced.

⚠

Warning: The test set is touched exactly once — at the end, for the final evaluation. Snooping on it to tune choices is the Chapter 1 leakage sin in its most common real-world form, and it silently inflates every number you report.

6. Feature Scaling: Standardisation vs Normalisation

Age ranges 27–50; salary ranges 48,000–83,000. Any computation involving distances or gradients will be dominated by salary simply because its numbers are bigger — not because it matters more. Scaling puts features on a comparable footing. (Chapter 4 derives exactly how this distorts a linear model's coefficients; here we fix the mechanics.)

6.1 The two standard scalers

Standardisation: x′ = (x − x̄_train) / s_train

Centres on mean 0, SD 1 — Chapter 2's z-score, column by column. No fixed bounds; robust to outliers' influence on ranges; the default choice in this course.

Normalisation (min-max): x′ = (x − min_train) / (max_train − min_train)

Squeezes every column into [0, 1]. Intuitive and bounded — but a single extreme value compresses everything else; sensitive to outliers.

In practice both usually work; standardisation is the safer default because it does not cap values into a fixed interval (new data can legitimately fall outside the training range — min-max would mangle it, z-scores simply continue).

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train[:, 3:] = sc.fit_transform(X_train[:, 3:])   # learn train stats, apply to train
X_test[:, 3:]  = sc.transform(X_test[:, 3:])        # reuse train stats on test

Listing 3.7 — Scaling the numeric columns only (dummies stay untouched).

The column slice 3: skips the three country dummies and scales only Age and Salary — a deliberate choice explained below.

6.2 Which algorithms need scaling?

Needs scaling	Why	Chapters
K-Nearest Neighbors	Pure distance computation	12
SVM / Kernel SVM / SVR	Margins and kernels are geometric	7, 13, 14
Neural networks (ANN, CNN)	Gradient descent converges faster on comparable scales	26, 27
PCA & Kernel PCA	Variance is scale-dependent	28, 30
K-Means	Cluster distances	19
Logistic / linear regression (gradient-solvers)	Faster, stabler convergence	4, 5, 11

Naturally scale-free	Why	Chapters
Decision trees	Splits compare one feature at a time to a threshold — units cancel	8, 16
Random forests	Averages of scale-free trees	9, 17
Boosted trees (XGBoost)	Same threshold logic	32
Naive Bayes	Per-feature likelihoods handle scale internally	15

💡

Pro Tip — the dummy-variable question: Should one-hot columns be scaled too? Both schools exist. This course (like the source code) leaves dummies unscaled: a 0/1 column standardised loses its clean interpretation, and the dummy's information is not distorted by units. Scaling them is not wrong — it is a different convention; just be consistent.

📋 Stable content — Reviewed: August 2026

7. The Correct Order — Beating Data Leakage

Here is the chapter's most professional lesson, and an honest critique of the otherwise-excellent source code. In data_preprocessing_tools.py, the imputer and the encoder are fitted on the full dataset before the split. The imputed mean of Salary, for instance, is computed from all ten rows — including the two that later land in the test set. The test set has quietly leaked information into training. The leakage is mild (a mean over two extra rows), but the pattern matters enormously: in production pipelines the same mistake with scalers, encoders or feature selectors can turn a good model into a lying one.

The leakage-proof rule is one sentence:

⚠

Fit preprocessing on the training set only; transform both sets with what was learned. Any statistic computed on the full dataset — mean, min, max, categories, vocabulary — before the split hands the test set a vote in its own exam.

Figure 3.2 — The leakage-proof order: every transformer learns from the training set; the test set only ever receives transform().

sklearn's Pipeline bundles these steps so the fit/transform boundary is enforced automatically — you fit the whole pipeline on the training rows and cannot accidentally leak. Pipelines get their full treatment (with ColumnTransformer for mixed-type columns) in the Chapter 33 capstone; Section 8 below shows the correct manual order in the meantime.

8. The Complete Preprocessing Template

The full pipeline, modernised and re-ordered leakage-proof — this is the template every later chapter starts from:

# --- Data Preprocessing Template (leakage-proof) ---
import numpy as np
import pandas as pd

# 1. Load and split features / target
dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

# 2. Encode the target
from sklearn.preprocessing import LabelEncoder
y = LabelEncoder().fit_transform(y)

# 3. Split FIRST — the test set must not influence any statistic
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1)

# 4. Impute — learn from train only
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
X_train[:, 1:3] = imputer.fit_transform(X_train[:, 1:3])
X_test[:, 1:3]  = imputer.transform(X_test[:, 1:3])

# 5. One-hot encode the nominal feature — learn categories from train only
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(
    transformers=[('encoder', OneHotEncoder(sparse_output=False,
                                            handle_unknown='ignore'), [0])],
    remainder='passthrough')
X_train = np.array(ct.fit_transform(X_train))
X_test  = np.array(ct.transform(X_test))

# 6. Scale the numeric columns — train statistics only
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train[:, 3:] = sc.fit_transform(X_train[:, 3:])
X_test[:, 3:]  = sc.transform(X_test[:, 3:])

Listing 3.8 — The complete, leakage-proof preprocessing template.

The preprocessing ritual (the checklist professionals run)

info() + describe() — types, counts, missing census (Chapter 2's audit)
Split X (features) and y (target); drop rows with missing y
Decide the missing-data mechanism and fix: delete (rarely) or impute (mean / median / mode / constant + indicator)
Encode: one-hot nominal features, label the target; know the dummy trap
Split train/test with a fixed random_state (stratify for imbalanced classification)
Scale if the algorithm is distance- or gradient-based — fit on train, transform both
Document every choice above: your future self is a different person

🌎

Real World: In production systems, this entire template lives inside a fitted Pipeline object, saved to disk with joblib and applied verbatim to every new batch of raw data — guaranteeing the transformations applied tomorrow match the ones used in training. Chapter 33 builds exactly that.

🛠 Hands-On: Preprocess the Startups DatasetBeginner

You are handed 50_Startups.csv — 50 US startups with R&D Spend, Administration, Marketing Spend, State (New York / California / Florida) and Profit. The goal (Chapter 5 will model it): predict Profit from the rest. Apply the full preprocessing ritual — leakage-proof order — and answer the interpretation questions.

Steps

Load the data; run the audit (info(), describe(), isnull().sum()). Any gaps? Any dtype surprises?
Split into X (4 features) and y (Profit).
Split into train/test: 20% test, random_state=42. Why is stratify unnecessary here?
One-hot encode State with a ColumnTransformer — fitted on train only. How many dummy columns emerge, and why exactly that many?
Standardise the numeric spend columns — train statistics only. Do you scale the dummies? State the convention and the reason.
Print the shapes before and after encoding. Explain the change.

View Solution / Walkthrough

import numpy as np
import pandas as pd

dataset = pd.read_csv('50_Startups.csv')
X = dataset.iloc[:, :-1].values          # R&D, Admin, Marketing, State
y = dataset.iloc[:, -1].values           # Profit

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

ct = ColumnTransformer(
    transformers=[('encoder', OneHotEncoder(sparse_output=False,
                                            handle_unknown='ignore'), [3])],
    remainder='passthrough')
X_train = np.array(ct.fit_transform(X_train))
X_test  = np.array(ct.transform(X_test))
print('Shape after encoding:', X_train.shape)    # (40, 6)

sc = StandardScaler()
X_train[:, 3:] = sc.fit_transform(X_train[:, 3:])
X_test[:, 3:]  = sc.transform(X_test[:, 3:])

Answers:

Audit: 50 rows, no missing values, and all numeric columns load as float — no repair needed; the only categorical is State.
Stratify: unnecessary because the target Profit is continuous — stratification preserves class proportions and only applies to classification targets.
Dummies: three — one per observed state (New York, California, Florida). (Chapter 5 will drop one of them to dodge the dummy trap; the encoding mechanics are identical.)
Shape: 40 training rows × 4 columns become 40 × 6 — State expands from one column to three dummies; the three spend columns pass through untouched.
Dummies scaled? No — this course's convention keeps 0/1 columns unscaled to preserve their clean interpretation; the three spend columns (means around six figures) are standardised because Chapter 5's model will compare their coefficients.

🎯 Key Takeaways

Models see a numeric matrix, not your problem. Preprocessing — repair, translate, divide, rescale — decides whether that matrix teaches truth or distortion.

The missing-data mechanism (MCAR / MAR / MNAR) dictates the fix: mean for symmetric numerics, median for skewed, mode for categoricals — and never impute the target.

One-hot nominal features (no fake order or distance); label-encode the target (no geometry involved); know the dummy trap: drop one category for linear models.

Scale for distance- and gradient-based algorithms (K-NN, SVM, ANN, PCA, K-Means); trees and forests are naturally immune — and this course leaves dummy columns unscaled.

Split before you fit anything: every statistic (means, categories, scaler parameters) is learned from the training set and merely applied to the test set.

Fix random_state for reproducibility, stratify for imbalanced classification, and document every preprocessing choice — the ritual is the profession.