Descriptive Statistics & Outliers
The numbers that summarize a dataset — center, spread, and shape — and how to spot the outliers that distort them.
Learning Objectives
- Distinguish descriptive vs inferential statistics.
- Explain mean, median, mode — and when to use each.
- Explain range, variance, standard deviation, and IQR.
- Understand skewness and its effect on the mean vs median.
- Detect outliers using the IQR and z-score methods.
1. Descriptive vs Inferential Statistics
| Descriptive | Inferential | |
|---|---|---|
| Purpose | summarize & describe the data you have | draw conclusions about a population from a sample |
| Examples | mean, median, charts | hypothesis tests, confidence intervals |
This session focuses on descriptive statistics — the "what does my data look like?" toolkit.
2. Mean, Median, Mode — the Center
| Measure | What it is | Best when |
|---|---|---|
| Mean | average (sum ÷ count) | data is roughly symmetric, no big outliers |
| Median | middle value | data is skewed or has outliers |
| Mode | most frequent value | categorical data |
import numpy as np
data = [10, 20, 30, 40, 1000] # 1000 is an outlier
np.mean(data) # 220 — pulled way up by the outlier
np.median(data) # 30 — robust, ignores the outlier3. Measures of Spread
| Measure | What it is |
|---|---|
| Range | max − min |
| Variance | average of squared deviations from the mean |
| Standard deviation | square root of variance — spread in the same units as the data |
| IQR | Q3 − Q1 (middle 50% of data) |
np.std(data) # standard deviation
np.var(data) # varianceStandard deviation is the most-used: a small SD means data clusters tightly around the mean; a large SD means it's spread out.
4. Skewness
Skewness describes whether a distribution leans to one side:
- Right-skewed (positive) — a long tail to the right; mean > median.
- Left-skewed (negative) — a long tail to the left; mean < median.
- Symmetric — mean ≈ median.
Income data is a classic right-skewed example — a few very high earners pull the mean above the median.
5. Outliers — What They Are
An outlier is a value that is significantly different from the rest — far enough to distort your statistics. Detecting them is a core analyst skill.
Method 1 — IQR (Interquartile Range)
Q1 = np.percentile(data, 25)
Q3 = np.percentile(data, 75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = [x for x in data if x < lower or x > upper]Any value below Q1 − 1.5×IQR or above Q3 + 1.5×IQR is an outlier. The IQR method is robust because it uses quartiles, not the mean.
Method 2 — Z-score
mean = np.mean(data)
std = np.std(data)
z_scores = [(x - mean) / std for x in data]
outliers = [x for x, z in zip(data, z_scores) if abs(z) > 3]A z-score tells you how many standard deviations a value is from the mean. A common threshold is |z| > 3.
6. Handling Outliers
Once detected, you have options — and the right one depends on context:
- Investigate first — is it a data-entry error or a real, meaningful extreme?
- Remove — if it's a genuine error.
- Cap / winsorize — replace with a threshold value.
- Transform — log transform to reduce the effect of extremes.
- Keep — if it's real and important (e.g., a legitimately huge customer).
7. Interview Questions (with Model Answers)
The descriptive statistics questions interviewers ask. Self-test before revealing.
IQ1. What's the difference between descriptive and inferential statistics?
Model answer: "Descriptive statistics summarize the data you have — mean, median, charts. Inferential statistics draw conclusions about a larger population from a sample — like hypothesis tests."
IQ2. What's the difference between mean, median, and mode, and when do you use each?
Model answer: "Mean is the average, median the middle value, mode the most frequent. I use mean for symmetric data, median when data is skewed or has outliers, and mode for categorical data."
IQ3. Why is the median more robust to outliers than the mean?
Model answer: "Because the mean sums every value, one extreme can drag it far away. The median only cares about the middle position, so outliers barely affect it."
IQ4. Explain variance and standard deviation.
Model answer: "Both measure spread. Variance is the average of squared deviations from the mean; standard deviation is its square root, so it's in the same units as the data and easier to interpret."
IQ5. What is skewness?
Model answer: "Skewness measures asymmetry. Right-skewed data has a long right tail and the mean above the median; left-skewed has the opposite. Income is a classic right-skewed example."
IQ6. What is an outlier, and how do you detect one?
Model answer: "An outlier is a value far from the rest that can distort statistics. I detect them with the IQR method (values beyond Q1−1.5×IQR or Q3+1.5×IQR) or z-scores (|z| > 3), plus boxplots visually."
IQ7. Explain the IQR method for outlier detection.
Model answer: "Compute Q1 and Q3, then IQR = Q3 − Q1. Any value below Q1 − 1.5×IQR or above Q3 + 1.5×IQR is flagged. It's robust because it's based on quartiles, not the mean."
IQ8. Explain the z-score method for outlier detection.
Model answer: "A z-score is how many standard deviations a value is from the mean. Values with |z| above a threshold (often 3) are outliers. It assumes roughly normal data, since the mean and SD can be distorted by outliers themselves."
IQ9. How do you handle outliers once you find them?
Model answer: "First I investigate whether it's an error or a real extreme. Then I might remove it, cap it, or log-transform — or keep it if it's meaningful. I never delete blindly."
Hands-On Project: Describe and Detect Outliers
Using a list of sales values, compute descriptive statistics and detect outliers.
Steps
- Create
sales = [100, 120, 110, 130, 115, 900](900 is an outlier). - Compute the mean and median.
- Compute the standard deviation.
- Find outliers using the IQR method.
- Find outliers using the z-score method (|z| > 3).
- Explain which method is better here and why.
View Solution / Walkthrough
import numpy as np
sales = [100, 120, 110, 130, 115, 900]
print(np.mean(sales)) # 245.8 — pulled up by 900
print(np.median(sales)) # 117.5 — robust
print(np.std(sales)) # large, due to the outlier
# IQR method
Q1, Q3 = np.percentile(sales, 25), np.percentile(sales, 75)
IQR = Q3 - Q1
lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR
iqr_outliers = [x for x in sales if x < lower or x > upper] # [900]
# z-score method
mean, std = np.mean(sales), np.std(sales)
z_outliers = [x for x in sales if abs((x - mean) / std) > 3] # [900]Which is better? With a single big outlier, the IQR method is more reliable — the z-score method's mean and SD are themselves distorted by the 900, which can mask the outlier.
Key Takeaways
Mean is sensitive to outliers; median is robust — use median for skewed data.
Standard deviation = spread in the data's own units.
Right-skewed → mean > median; left-skewed → mean < median.
IQR method is robust; z-score assumes roughly normal data.
Never delete outliers blindly — investigate the cause first.
Objective Questions — Test Your Understanding
Q1. Which measure of central tendency is most affected by outliers?
Q2. Which measure is most robust to outliers?
Q3. What does standard deviation measure?
Q4. In the IQR method, an outlier is above Q3 + 1.5×IQR or below…
Q5. What does a z-score of +2 mean?