Mini-Project 2 · Will the Loan Default? — an ANN for Credit Risk

1. The Brief

You are the first analyst at a small consumer lender. The bank's rule today is blunt: approve everyone with a credit score above 600. The credit desk wants better. Three questions:

• Question A (ANN): can Chapter 26's network — a small stack of Dense layers — learn default risk from five applicant numbers?
• Question B (the race): does the network beat logistic regression? And what happens when a human who knows credit adds two engineered features by hand?
• Question C (the money): a repaid loan earns $2,000 in interest; a default loses $8,000. Where should the approval threshold sit, and what is it worth per 300 applicants?

Question A is Chapter 26 again, on new numbers. Question B is the chapter's oldest rivalry, made fair. Question C is where machine learning meets the actual business — Chapter 26 §6.1's "the threshold is a business decision", with dollars attached.

2. The Dataset

The corpus is synthetic — generated from a credit-risk recipe with a fixed random seed, so every student downloads the exact same 1,500 rows and the solution's numbers reproduce on your machine. It is honest about being synthetic: five features and one deliberately learnable pattern. Real bureaus feed models hundreds of variables. Treat this as a flight simulator.

Two real rows from the file — the two ends of the risk scale (you will meet them again in the solution):

The generator's one secret (revealed and re-derived in the solution): risk is driven by the burden — loan divided by income — by the credit score, and by one interaction between them: a heavy burden is survivable with good credit, and mild with any credit, but heavy burden plus weak credit is where defaults concentrate. That interaction is exactly the kind of pattern a linear model cannot see in raw columns — and a network can learn. Question B will test whether it does.

3. Your Job

1. Explore. Load the TSV. Default rate? Feature ranges? Any surprising values?
2. Split. 80/20, random_state=0, stratify=y — the default class is 21%, protect it.
3. Scale — carefully. StandardScaler, fit on the training set only, then transform both sets. Say in one sentence why the network needs this and why the fit must not see the test set.
4. Baseline. Logistic regression on the five scaled features. Record accuracy, AUC, confusion matrix.
5. The ANN. Keras Sequential: Dense(8, relu) → Dense(4, relu) → Dense(1, sigmoid); compile with adam and binary cross-entropy; 100 epochs, batch 32; set_random_seed(0). Same metrics, same test set.
6. The race, round two. Engineer two features by hand — the burden, and the interaction max(0, burden − 0.42) × max(0, 600 − score) — add them to the raw five, and rerun logistic. Full scoreboard: who wins, and what does that teach about when a network earns its complexity?
7. The money. Approve an applicant when P(default) < t. With G = $2,000 and L = $8,000, sweep t from 0.05 to 0.95 and plot profit. Compare the optimum with the formula p* = G / (G + L). Compute the profit of approve-all and reject-all. Write the desk a three-line recommendation.

4. Solution & Real Results

Work the steps first. Then read the code walkthrough below — every block explained, with special attention to the scaler (the step students break most often) and the money block. Every number is from one seeded, deterministic run on the file you downloaded.

4.1 The code, line by line

Block 1 · Load and look.

ds = pd.read_csv('loan_applications.tsv', sep='\t')
print(len(ds), ds['Default'].mean())
# 1500  0.21   — one in five loans defaults

1,500 rows, 21% default rate. That single number sets the agenda: the classes are imbalanced (roughly 4:1), so plain accuracy has a comfortable floor (79% by predicting "repaid" for everyone) — you will need the confusion matrix and AUC, and the split needs stratify.

Block 2 · The split, with the minority protected.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y)   # 1200 / 300

Same discipline as Mini-Project 1: 1,200 applications to learn from, 300 to be judged on, and stratify=y keeps the 79:21 class ratio in both — the test set lands at 237 repaid / 63 defaulted. random_state=0 fixes the shuffle so every number below reproduces.

Block 3 · The scaler — the step everyone breaks.

sc = StandardScaler()
X_train_s = sc.fit_transform(X_train)   # learn mean/std on TRAIN only
X_test_s  = sc.transform(X_test)        # apply the SAME statistics

Two questions answered here. Why scale at all? Income lives in the tens of thousands; MonthsEmployed in the hundreds. Unscaled, one weight multiplies huge numbers and another multiplies tiny ones — gradient descent struggles, exactly as Chapter 26 explained. Scaled, every feature has mean 0 and standard deviation 1, and the weights start on a level field. Why fit on the training set only? The scaler is a model — it learns two numbers per feature. Fitting it on all 1,500 rows lets test-set information leak into training. Chapter 3's rule, now with teeth: fit_transform the train set, transform the test set. Never the other way around.

Block 4 · The baseline.

lr = LogisticRegression(max_iter=2000).fit(X_train_s, y_train)
p_lr = lr.predict_proba(X_test_s)[:, 1]      # P(default) per applicant

Chapter 11's model, scaled and honest. On the raw five features it scores 0.900 accuracy / 0.942 AUC — a strong baseline. Note the habit from Chapter 18: take predict_proba, not just the class, because the money block later needs the probabilities.

Block 5 · The network, line by line.

tf.keras.utils.set_random_seed(0)
model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')])
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(X_train_s, y_train, epochs=100, batch_size=32, verbose=0)
p_ann = model.predict(X_test_s, verbose=0).ravel()

Read it as three decisions. Architecture (the three Dense lines): the first layer turns 5 inputs into 8 weighted sums, relu-clipped; the second compresses 8 → 4; the sigmoid output squeezes to one probability — Chapter 26's 12-6-6-1 churn net, shrunk to 8-4-1. Count with §2.1's arithmetic: 5·8+8 + 8·4+4 + 4·1+1 = 89 parameters. Compile: adam (the gradient-descent upgrade) and binary cross-entropy — the loss whose gradient is the logistic chapter's old friend. Fit: 100 passes over 1,200 rows in batches of 32 — about 3,700 weight updates. ravel() flattens the (300, 1) prediction into a flat vector, so it compares cleanly with y_test.

Block 6 · The scoreboard.

# accuracy / AUC on the same 300 test applicants
LR raw      : 0.9000 / 0.9422      cm [[221 16] [14 49]]
ANN (8-4-1) : 0.9100 / 0.9445      cm [[225 12] [15 48]]
LR + engineered : 0.9267 / 0.9508  ← the winner

Round one: the network beats raw logistic — 0.910 vs 0.900 accuracy, 0.9445 vs 0.9422 AUC. The margin is real but small. The reason is the interaction: the network can bend its decision surface where burden and weak credit cross; the linear model cannot. Round two is the humbling one: give logistic two hand-made features — the burden itself and the interaction term — and it wins the whole race at 0.927 / 0.951. The network's advantage was never magic. It learned in 3,700 updates what a credit analyst could write down in two columns — because here we knew what to write. When you don't know the interactions, don't have a domain expert, or have hundreds of features — that is when the network earns its keep.

Block 7 · The engineered features.

burden = X[:, 2] / X[:, 1]                                # loan / income
weak   = np.maximum(0, 600 - X[:, 3]) / 100               # weak-credit depth
X_eng  = np.column_stack([X, burden,
                          np.maximum(0, burden - 0.42) * weak])   # the interaction

Two lines of domain knowledge. The burden says how heavy the loan is for this income. The interaction fires only when burden exceeds 0.42 and the score is below 600 — the corner where defaults concentrate (22% of applicants sit in it). Feed these to logistic and its linear boundary is enough again. The honest ranking to report: knowing the domain beat the network; the network beat not knowing.

Block 8 · The money.

G, L = 2000, 8000                    # interest earned / loss on default
profit_t = []
for t in np.arange(0.05, 1.0, 0.05):
    approve = p_ann < t              # lend when P(default) is low enough
    profit = G*((approve) & (y_test==0)).sum() - L*((approve) & (y_test==1)).sum()
    profit_t.append(profit)

The rule: approve when P(default) < t. Every approved repayer pays $2,000; every approved defaulter costs $8,000. Sweep the dial and the real numbers come out:

Three readings. First, the dumb rules fail: approving everyone loses $30,000 — at 21% defaults and a 4:1 loss ratio, lending blindly is negative. Second, the formula: approve while expected profit is positive, (1−p)·G − p·L > 0, which rearranges to p < G/(G+L) = 0.20 — Chapter 33's t* = offer/CLV, reborn as credit policy. Third, the sweep disagrees with the formula: the empirical optimum sits at 0.35 worth $352,000. Why? The formula assumes perfectly calibrated probabilities. An 89- parameter network on 1,200 rows is close, not perfect — its "0.35" behaves like a true 0.20 on this sample. The lesson is not to choose one over the other: the formula tells you where to look, the sweep tells you what the model actually does. In production you would recalibrate (Chapter 18) and re-sweep.

Block 9 · Reading two applicants.

# from the canonical run — the two ends of the scale
safest  : p = 0.000   Age 26 · Income $126,811 · Loan $14,997 · Score 817 → repaid
riskiest: p = 1.000   Age 55 · Income $25,692  · Loan $36,814 · Score 323 → defaulted

Sanity-check the network's voice. The safest applicant carries a trivial burden (0.12) and a near-perfect score — the network is rightly certain. The riskiest owes 1.43× their income with a 323 score — the interaction corner, and the loan did default. A model whose extremes you can explain is a model you can defend to a regulator.

6. The Honest Limits

• Five features. Real bureaus feed hundreds; the interaction here is one clean corner. Real risk surfaces are messier, and the ANN-vs-engineering race shifts with every extra feature.
• The labels are Bernoulli draws. A real "default" is months of collections, restructuring and judgement calls — not one random draw from a formula.
• Calibration was assumed, then measured. The formula-vs-sweep gap (0.20 vs 0.35) is this sample's truth; a production system would calibrate probabilities (Chapter 18) before trusting any threshold.
• No time, no macro. The file has no recession, no rate hikes, no drift. A model trained in good times is a good-times model.
• No fairness audit. Deliberately out of scope here — never out of scope in real lending.