Quantitative Analysis II — Regression & ML Evaluation

Learning Objectives

By the end of this session, you will be able to:

• Specify, estimate, and interpret a multiple linear regression model — including unstandardised and standardised coefficients, R² and adjusted R², significance of individual predictors, and the overall model F-test
• Diagnose regression models for violations of assumptions — linearity, normality of residuals, homoscedasticity, independence, multicollinearity, and influential cases — and apply appropriate remedies when assumptions are violated
• Apply and interpret binary logistic regression — including odds ratios, model fit (Hosmer-Lemeshow, pseudo-R²), classification accuracy, and the distinction between statistical significance and predictive utility
• Select and compute appropriate ML evaluation metrics — accuracy, precision, recall, F1-score, ROC-AUC — based on the problem type, class distribution, and the relative cost of false positives vs. false negatives
• Design and implement a rigorous evaluation protocol including k-fold cross-validation, baseline comparisons, and statistical tests for model comparison (e.g., McNemar's test) that supports valid conclusions about model performance

Session Planner

Suggested breakdown of the 4-hour contact session.

Time	Segment	Activity	Mode
0:00–0:08	Opening	Recap Week 13; transition: "Last week: single-variable relationships and group differences. This week: models with multiple predictors, binary outcomes, and systematic evaluation."	Whole class
0:08–0:30	Lecture 1	Multiple linear regression — the regression equation, interpreting coefficients (b and β), R² and adjusted R², hierarchical vs. simultaneous entry, the F-test for overall model and t-tests for individual predictors	Lecture
0:30–0:50	Lecture 2	Regression diagnostics — linearity, normality of residuals, homoscedasticity, independence (Durbin-Watson), multicollinearity (VIF, tolerance), outliers and influential cases (Cook's distance). Logistic regression — when to use it, odds ratios, model fit, classification tables.	Lecture
0:50–1:05	Activity	Regression output interpretation: given SPSS/R output tables, extract and correctly interpret coefficients, significance, R², and diagnostics. Identify which assumptions appear violated from diagnostic plots.	Pairs
1:05–1:15	Discussion	Debrief interpretation exercise; address common errors (confusing b and β, ignoring VIF, overinterpreting R²)	Whole class
1:15–1:30	Break	—	—
1:30–1:55	Lecture 3	ML evaluation — accuracy and its limitations (imbalanced data); precision, recall, F1; the confusion matrix; ROC curves and AUC; k-fold cross-validation; statistical comparison of models (McNemar's test); benchmarking against baselines	Lecture
1:55–2:10	Activity	Metric selection: given 8 ML scenarios with different class distributions and error costs, select the most appropriate evaluation metric(s) and justify	Pairs
2:10–3:30	Lab Work	BBA: Run and diagnose multiple regression on capstone/secondary data. BCA: Compute confusion matrix and evaluation metrics; implement cross-validation; compare models.	Individual
3:30–3:50	Peer Review	Exchange regression diagnostics / ML evaluation outputs; peer checks for correct interpretation and missing diagnostics	Pairs
3:50–4:00	Exit Ticket	Submit analysis output with interpretation; identify the most concerning diagnostic or the weakest evaluation metric	Individual

1. Multiple Linear Regression

Multiple regression extends the logic of correlation (Week 13) to models with multiple predictors. While correlation tells you whether two variables are related, regression tells you: (a) how much of the variance in the outcome is collectively explained by the predictors, (b) which predictors uniquely contribute to the model, (c) the direction and magnitude of each predictor's unique effect while controlling for all other predictors, and (d) the predicted value of the outcome for any given combination of predictor values. This conditional, multi-predictor logic is what makes regression the workhorse of quantitative social science and business research.

1.1 Key Output to Interpret

Output	What It Tells You	How to Interpret	APA Report
R²	The proportion of variance in the DV explained by the model — an overall measure of model fit	R² = .42 means the model explains 42% of variance in the DV. Adjusted R² is preferred — it penalises for adding predictors that don't improve the model, preventing overfitting inflation.	"The model explained 42% of the variance in job satisfaction, R² = .42, adjusted R² = .40, F(4, 182) = 32.94, p < .001."
F-test (ANOVA table)	Whether the model as a WHOLE explains significantly more variance than the mean alone (the intercept-only model)	If p < .05, the model with all predictors is a significant improvement over no model. If p > .05, the model does not significantly predict the DV — stop here; individual predictors should not be interpreted.	Report alongside R² as shown above. F(dfregression, dfresidual) = F-value.
b (unstandardised coefficient)	The change in the DV (in original units) for a one-unit change in the predictor, holding all other predictors constant	b = 0.34 for "autonomy" (1–5 scale) predicting job satisfaction (1–5 scale): a one-point increase in autonomy is associated with a 0.34-point increase in satisfaction, controlling for other predictors.	"Autonomy was a significant positive predictor, b = 0.34, 95% CI [0.21, 0.47], β = .31, t(182) = 5.23, p < .001."
β (standardised coefficient)	The change in the DV (in SD units) for a one-SD change in the predictor. Enables comparison of relative importance across predictors measured on different scales.	β = .31 for autonomy, β = .42 for work-life balance. Work-life balance has a stronger unique effect on job satisfaction than autonomy, even if both are significant. Compare |β| values — but never claim one is "significantly larger" without formal comparison.	Always report both b and β. β is for comparing predictors within a study. b is for comparing across studies or making practical predictions.
t-test (per predictor)	Whether each individual predictor contributes significantly to the model, AFTER controlling for all other predictors	p < .05 means the predictor adds significant unique variance beyond what the other predictors already explain. A non-significant predictor may still be correlated with the DV — it just doesn't add UNIQUE variance beyond the other variables in the model.	Report t, df, and p for each predictor as shown above.

1.2 Hierarchical vs. Simultaneous Entry

Approach	How It Works	When to Use	Key Output
Simultaneous (Enter)	All predictors are entered into the model at once. Each predictor's contribution is assessed controlling for all others.	When you have no theoretical reason to prioritise certain predictors; when the goal is to identify which variables uniquely predict the DV; the default in most capstone research	One model: R², F-test, and coefficients for all predictors. The simplest approach.
Hierarchical (Block Entry)	Predictors are entered in blocks/steps based on theory. Block 1: control variables (demographics). Block 2: main predictors. Block 3: interaction terms. At each step, the R² change (ΔR²) is tested for significance.	When theory specifies a causal or temporal ordering of predictors; when you want to know whether Block 2 variables explain variance BEYOND Block 1 controls; when testing mediation (Blocks → Baron & Kenny approach)	Multiple models: R² at each step, ΔR² and its significance. "Adding the three work characteristics in Step 2 explained an additional 18% of variance, ΔR² = .18, ΔF(3, 179) = 24.61, p < .001."

2. Regression Diagnostics — Checking Whether Your Model is Valid

Regression makes assumptions, and violating them produces misleading coefficients, incorrect standard errors, and invalid significance tests. Diagnostics are not optional post-hoc checks — they are an integral part of regression analysis. A regression model that has not been diagnostically checked is an untrustworthy model.

2.1 The Six Assumptions and How to Check Them

Assumption	What It Means	How to Check	Remedy if Violated
1. Linearity	The relationship between each predictor and the DV is linear. The model cannot capture non-linear relationships (unless transformed).	Scatterplots of each predictor vs. DV; residual vs. fitted plot (should show no systematic pattern — a random cloud is what you want). LOESS curves overlaid on scatterplots.	Transform the predictor (log, square root, polynomial term); add a quadratic term (X²); consider non-linear regression or GAM if relationships are complex.
2. Normality of Residuals	The residuals (errors) are approximately normally distributed. This assumption is about the residuals, NOT the raw variables.	Histogram of residuals; Q-Q plot of residuals; Shapiro-Wilk test on residuals. With n > 100, the CLT makes regression robust to moderate non-normality.	If severe: transform the DV (log, square root, Box-Cox); use robust regression (bootstrapped standard errors); consider a non-parametric alternative.
3. Homoscedasticity	The variance of residuals is constant across all levels of the predicted values. The spread of residuals should not fan out or narrow systematically.	Residual vs. fitted plot: the vertical spread should be roughly constant. Breusch-Pagan test. If the plot shows a funnel shape (wider at one end), heteroscedasticity is present.	Transform the DV; use heteroscedasticity-consistent standard errors (HC3 or HC4; also called "robust standard errors"); weighted least squares regression.
4. Independence of Errors	Residuals are independent of each other. The error for one observation does not predict the error for another. Critical for time-series or grouped data.	Durbin-Watson statistic: values close to 2 indicate no autocorrelation. Values < 1 or > 3 are problematic. Also: think about your design — are observations nested or clustered?	For time-series: use time-series models (ARIMA). For nested data: use multilevel modelling / mixed effects models. Clustered standard errors for grouped data.
5. No Multicollinearity	Predictors are not too highly correlated with each other. High multicollinearity inflates standard errors, making it impossible to separate predictors' unique effects.	VIF (Variance Inflation Factor): VIF < 5 is acceptable. VIF > 10 is a serious problem. VIF > 5 with a non-significant predictor suggests the predictor's contribution is masked by collinearity. Tolerance: 1/VIF; values < 0.2 (VIF > 5) are concerning.	Remove one of the highly correlated predictors (keep the theoretically more important one); combine correlated predictors into a composite or factor score; use ridge regression or PCA regression.
6. No Influential Outliers	No single case or small number of cases disproportionately influences the regression coefficients. One extreme case can pull the regression line substantially.	Cook's Distance: Values > 1 are problematic; values > 4/n warrant investigation. Leverage: values > 2(k+1)/n are high leverage. Standardised residuals: values > |3| are potential outliers.	Investigate influential cases. If data entry errors: fix. If genuine but extreme: run analysis with and without the case(s); report both and discuss sensitivity. Never silently delete cases.

2.2 Reporting Regression in APA Format

3. Binary Logistic Regression — When Your Outcome is Yes/No

Linear regression requires a continuous, unbounded DV. When your outcome is binary — purchase (yes/no), churn (yes/no), pass (yes/no) — linear regression produces nonsense (predicted probabilities outside 0–1, heteroscedastic errors, non-normal residuals). Logistic regression solves this by modelling the log-odds (logit) of the outcome as a linear function of the predictors, then transforming back to probabilities using the logistic function. The result: predicted probabilities between 0 and 1, interpretable odds ratios, and valid significance tests.

3.1 Key Output and Interpretation

Output	What It Tells You	Interpretation Example
Omnibus Tests (χ²)	Whether the model with predictors is significantly better than the intercept-only (null) model. Like the F-test in linear regression.	"The model was significant, χ²(4) = 47.32, p < .001, indicating that the predictors collectively distinguish between those who churned and those who did not."
Hosmer-Lemeshow Test	A goodness-of-fit test. p > .05 means the model fits the data adequately (the null is that the model fits — you WANT a non-significant result, unlike other tests).	"The Hosmer-Lemeshow test was non-significant, χ²(8) = 6.41, p = .60, indicating acceptable model fit." Note: this test is sensitive to sample size; with large n it may be significant despite decent fit.
Nagelkerke R²	A pseudo-R² approximating the proportion of variance explained. Not directly comparable to linear regression R². Typically lower.	"Nagelkerke R² = .34, suggesting the model explains approximately 34% of the variance in churn likelihood." Report as approximate — pseudo-R² values are informative but not precise.
Classification Table	Shows what percentage of cases the model correctly classifies, at a default 0.5 probability cutoff. Includes sensitivity and specificity.	"The model correctly classified 78.2% of cases overall, with 82.1% of non-churners and 68.4% of churners correctly identified." Report both sensitivity and specificity — overall accuracy can be misleading with imbalanced classes.
Odds Ratio (Exp(B))	The multiplicative effect on odds. OR = 1.0: no effect. OR = 2.0: odds double per unit increase. OR = 0.5: odds halve per unit increase.	"Satisfaction was a significant predictor (OR = 0.62, 95% CI [0.48, 0.80], p = .001). For each one-point increase in satisfaction, the odds of churning decreased by 38% (i.e., multiplied by 0.62)." Always report 95% CI for OR.

4. Machine Learning Evaluation — Beyond Accuracy

BCA capstones involving classification models require a fundamentally different evaluation framework from the NHST-based approach covered in Weeks 13 and Sections 1–3. ML evaluation is concerned not with parameter inference (is β significantly different from zero?) but with predictive performance (how well does the model perform on unseen data?). Accuracy alone is rarely sufficient and is often misleading.

4.1 The Confusion Matrix — The Foundation of All Classification Metrics

4.2 The Core Metrics — When to Use Each

Metric	Formula	What It Captures	Best When...	Misleading When...
Accuracy	(TP + TN) / Total	Overall proportion correct — the simplest metric	Classes are balanced (roughly equal); the cost of FP ≈ cost of FN	Classes are imbalanced. A model that always predicts "no cancer" has 99% accuracy if cancer rate is 1% but is useless. Never report accuracy alone for imbalanced data.
Precision	TP / (TP + FP)	Of all positive predictions, how many were correct? Measures the "cost" of false positives.	False positives are costly. Email spam detection: flagging a genuine email as spam (FP) is worse than missing a spam email (FN).	You care equally about false negatives. Precision ignores false negatives entirely.
Recall (Sensitivity)	TP / (TP + FN)	Of all actual positives, how many did the model find? Measures the "cost" of false negatives.	False negatives are costly. Disease screening: missing a disease (FN) is far worse than a false alarm (FP). Fraud detection: missing fraud is worse than a false flag.	You care equally about false positives. A model that predicts "positive" for everything achieves 100% recall but zero precision.
F1-Score	2 × (Precision × Recall) / (Precision + Recall)	The harmonic mean of precision and recall. Balances the two when both matter.	You need a single metric that balances precision and recall; classes are moderately imbalanced; FP and FN have roughly similar costs.	The costs of FP and FN are very different. F1 weights them equally. Use weighted F1 (Fβ) if one error type matters more.
ROC-AUC	Area Under the ROC Curve	The model's ability to discriminate between classes across ALL possible thresholds. AUC = 1.0: perfect; 0.5: random; < 0.5: worse than random.	You want a threshold-independent measure of overall discriminative ability; comparing models globally; balanced or moderately imbalanced data.	Extremely imbalanced data (AUC can be optimistic); you need a metric at a specific operating point (AUC averages across all thresholds). Use PR-AUC for highly imbalanced data.

4.3 Cross-Validation — The Gold Standard for Model Evaluation

A single train/test split tells you how your model performs on ONE random partition of the data. A different split could produce a substantially different result. Cross-validation addresses this by repeating the process across multiple partitions and averaging the results, providing a more reliable estimate of generalisation performance.

Method	How It Works	Best For
k-Fold Cross-Validation	Split data into k equal folds. Train on k−1 folds, test on the held-out fold. Repeat k times (each fold serves as test once). Report mean and SD of metric across k folds.	The standard approach. k = 5 or 10 is typical. k = 10 provides a good bias-variance trade-off. Report: "Mean F1 = 0.82 (SD = 0.03) across 10-fold CV."
Stratified k-Fold	Like k-fold but preserves class proportions in each fold. Essential for imbalanced data.	Always use stratified CV for classification — it ensures each fold has roughly the same class distribution as the full dataset.
Repeated k-Fold	Repeat k-fold CV multiple times with different random splits. Further reduces variance of the performance estimate.	When you need a very stable performance estimate; when comparing models with small performance differences. 5 × 2 or 10 × 10 are common.
Leave-One-Out (LOO)	k = n: each observation serves as the test set once. Almost unbiased but very high variance.	Rarely used in capstone research due to computational cost. k = 10-fold is generally preferred for its lower variance.

4.4 Comparing Models — Beyond "My F1 is Higher"

Comparing model performance requires more than eyeballing mean metrics. A model may outperform another on one data split but underperform on another. Statistical tests for model comparison provide rigour.

Method	When to Use	What It Tells You
McNemar's Test	Comparing two classifiers on the SAME test data — examining whether the errors they make differ significantly	"The two models do not make the same errors; Model A correctly classified 23 cases that Model B got wrong, while Model B correctly classified only 8 that A got wrong. McNemar's χ²(1) = 7.26, p = .007 — Model A's advantage is significant."
Paired t-test on CV Folds	Comparing two models across k CV folds — testing whether the mean performance difference is significant	Run 10-fold CV for both models on the same folds. For each fold, compute the difference in F1. Paired t-test on the 10 differences. Report: t(9) = 3.41, p = .008, d = 1.08.
Baseline Comparison	ALL models should be compared against a meaningful baseline — random classifier, majority-class classifier, or published benchmark	"The proposed model achieved F1 = 0.82, compared with the majority-class baseline F1 = 0.12, the random baseline F1 = 0.48, and the published SOTA result F1 = 0.79 (Joshi, 2023)." Never report model performance in isolation — always with baselines.

5. From Analysis Plan to Results Chapter

5.1 BBA Analysis Workflow

5.2 BCA ML Evaluation Workflow

Think Deeper — Cross Questions

Discuss in pairs before sharing with the class.

Quick Check — Metric Selection

Select the most appropriate primary evaluation metric for each scenario.

Knowledge Check — Interactive Quiz

Test your understanding of regression and ML evaluation.

Lab Activity — Regression and ML Evaluation in Practice

Part A: Run and Diagnose a Multiple Regression (60 min)

1. Using your capstone data or the provided practice dataset, run a multiple regression with 3–6 predictors.
2. Complete the regression report table below.
3. Run all six diagnostics. For each: state the test/plot, the result, and whether the assumption is met.

Part B: Compute ML Evaluation Metrics (45 min)

1. Using your model's predictions or the provided sample output, construct the confusion matrix.
2. Compute: accuracy, precision, recall, F1, and (if applicable) ROC-AUC.
3. Identify the most appropriate PRIMARY metric for your problem. Justify with reference to class distribution and error costs.
4. Report your model's performance against at least two baselines.

Part C: Cross-Validation and Model Comparison (45 min)

1. Implement stratified k-fold CV (k = 5 or 10) for your model.
2. Report mean and SD of your primary metric across folds.
3. If comparing models: run CV for all models on the same folds. Compute the per-fold difference. Run a paired t-test or report the number of folds where Model A outperforms Model B.

Exit Ticket

Submit with your analysis output.

Key Takeaways — Week 14