Session 11: AI & Machine Learning Fundamentals

1. Machine Learning in Insurance — The Big Picture

Machine learning is not a magic wand. It is a set of statistical and computational techniques that find patterns in data and use those patterns to make predictions or decisions. In insurance, ML is valuable because the industry has exactly the characteristics that ML thrives on: large datasets, well-defined prediction problems, clear feedback loops, and significant financial impact from even small improvements in prediction accuracy.

1.1 Where ML Creates Value in Insurance

The impact ranges in the right-hand column are illustrative industry benchmarks drawn from McKinsey's AI-in-insurance research [4]; individual results depend heavily on data quality, product line, and implementation maturity.

1.2 The Insurance ML Maturity Curve

Most Indian insurers are at different stages of ML maturity. Understanding where an organisation sits on this curve is important for both internal strategy (what to invest in) and external analysis (what is a competitor capable of?):

• Level 0 — Descriptive (30% of insurers): Reports on what happened. SQL queries and dashboards. No predictive capability.
  Typical examples: smaller regional and cooperative insurers, and a few smaller private general insurers that still rely on basic MIS reports rather than analytics teams. (Small players are hard to name individually — this tier is defined by limited data infrastructure, not by brand.)
• Level 1 — Diagnostic (40% of insurers): Why did it happen? Online Analytical Processing (OLAP), basic segmentation, root-cause analysis with simple Excel models. Most public and mid-size private Indian insurers are here.
  Typical examples: the public-sector giants — LIC (life) and the four PSU general insurers (New India Assurance, National Insurance, United India Insurance, Oriental Insurance) — plus several mid-size private players. They have large data warehouses and Excel/OLAP-driven MIS, but predictive models are still point projects rather than embedded systems.
• Level 2 — Predictive (20% of insurers): What will happen? ML models deployed for specific use cases — fraud detection, churn prediction, claims severity. Large private insurers and digital carriers.
  Typical examples: HDFC Ergo, Bajaj Allianz, SBI General, HDFC Life, Max Life — large private insurers running production fraud, churn, and claims-severity models in specific lines; and digital carriers like Digit (automated claims triage).
• Level 3 — Prescriptive (8% of insurers): What should we do about it? ML models integrated into decision systems — automated underwriting, dynamic pricing, next-best-action for agents. Leading-edge private insurers.
  Typical examples: ICICI Lombard (AI-driven motor claims and the IL TakeCare app — models now steer underwriting and claims decisions), Max Life / HDFC Life (agent next-best-action and predictive underwriting in term plans), and Digit (automated claims settlement).
• Level 4 — Autonomous (2% of insurers): ML models drive decisions in real-time without human intervention for most standard cases. Straight-through processing for claims, fully automated underwriting. Only a few digital carriers (Acko, Lemonade) are here.
  Typical examples: Acko (India — 3-minute policy issuance via VAHAN API, straight-through processing for standard motor claims) and Lemonade (global — AI "Jim" handles claims end-to-end in seconds for simple cases).

Note on the percentages: the level shares (30% / 40% / 20% / 8% / 2%) are illustrative estimates intended to convey the rough distribution across the industry — they are not drawn from a single published survey. The framework itself (Descriptive → Diagnostic → Predictive → Prescriptive → Autonomous) follows Gartner's analytics maturity model [1]; real maturity data from McKinsey shows EMEA insurers' analytics-maturity scores span 18–85% [2], and Deloitte's global digital-maturity study is the complementary industry view [3]. See the References section at the end of this session.

2. Supervised vs. Unsupervised Learning

All machine learning falls into two broad categories. The distinction is simple: supervised learning has a target variable to predict; unsupervised learning does not. Each category contains multiple algorithm types suited to different insurance problems.

3. The ML Workflow

Every machine learning project in insurance follows the same six-step workflow. The details change — the features, the algorithm, the evaluation metric — but the structure is always the same. Mastering this workflow is more important than mastering any individual algorithm.

3.1 The Six-Step ML Workflow

┌─────────────────────────────────────────────────────────────────────┐
│                    THE ML WORKFLOW                                  │
│                                                                     │
│  ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐ │
│  │ 1. Problem  │   │ 2. Data    │   │ 3. Model   │   │ 4. Train   │ │
│  │ Definition  │──→│ Preparation│──→│ Selection  │──→│ & Validate │ │
│  └────────────┘   └────────────┘   └────────────┘   └────────────┘ │
│                                                         │          │
│  ┌────────────┐   ┌────────────┐                       │          │
│  │ 6. Deploy  │←──│ 5. Evaluate│◄───────────────────────┘          │
│  │ & Monitor  │   │            │                                    │
│  └────────────┘   └────────────┘                                    │
│                                                                     │
│  Feedback loop: Model predictions → Business decisions →           │
│  New data → Re-train → Better predictions                          │
└─────────────────────────────────────────────────────────────────────┘

3.2 The Scikit-Learn API Pattern

The Scikit-learn library provides a consistent API across all models. Once you learn this pattern, you can apply it to almost any algorithm with minimal code changes:

# The Universal Scikit-Learn Pattern:
#
# 1. Import the model class
# 2. Instantiate the model (with parameters)
# 3. Split data into training and test sets
# 4. Fit the model on the training data
# 5. Predict on the test data
# 6. Evaluate the predictions

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score, accuracy_score

# 1. Import
from sklearn.linear_model import LinearRegression

# Toy data so this pattern runs standalone (Section 4 replaces this with real
# insurance data). X = one feature per row, y = the target value to predict.
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]])
y = np.array([2.9, 6.1, 8.8, 12.2, 15.1, 18.0, 20.9, 24.1, 27.0, 30.2])  # ≈ 3x + noise

# 2. Instantiate
model = LinearRegression()

# 3. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size = 0.2, random_state = 0
)

# 4. Fit
model.fit(X_train, y_train)

# 5. Predict
y_pred = model.predict(X_test)

# 6. Evaluate
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)
print(f"RMSE: {rmse:.2f}")
print(f"R²: {r2:.3f}")

3.3 Train/Test Split — Why We Split

In insurance, the train/test split serves a crucial purpose: an ML model that performs well on the data it was trained on but poorly on new data is useless — and dangerous. The split simulates the real world: the model will be deployed on new policyholders and claims it has never seen. If it only learned the patterns in the training data (including its random noise), it will fail on new data. This is called overfitting, and we explore it in Section 7.

# Always split before training — never train on your test data!
# Common splits: 80/20 (large datasets), 70/30 (smaller), 60/20/20 (train/val/test)

X_train, X_test, y_train, y_test = train_test_split(
    X, y,                # X, y are the toy data defined in §3.2 above
    test_size=0.2,       # Hold back 20% for testing
    random_state = 0      # For reproducible results
)

# For classification (categorical target), add stratify=y to preserve class
# proportions in each split:
# X_train, X_test, y_train, y_test = train_test_split(
#     X, y, test_size = 0.2, random_state = 0, stratify = y
# )

4. Regression — Predicting Claim Amounts

Regression predicts a continuous numeric value. In insurance, the most common regression task is predicting claim amounts — given policyholder characteristics, what is the expected claim amount? This is used for: pricing (what premium should we charge?), reserving (how much money should we set aside?), and underwriting (which applications should we accept?).

4.1 Preparing Data for Regression

# Preparing Data for Regression

# Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('data/insurance_cleaned.csv')

# Keep only claims with a known amount, and drop rows with missing features
dataset = dataset[dataset['claim_amount'].notna()].copy()
feature_cols = ['age', 'income', 'credit_score', 'premium', 'sum_assured', 'policy_type']
feature_cols = [c for c in feature_cols if c in dataset.columns]
dataset = dataset.dropna(subset=feature_cols)

# Separate features and target (X = features, y = claim amount to predict)
X = dataset[feature_cols]          # keep as DataFrame so ColumnTransformer can use column names
y = dataset['claim_amount'].values

# Identify categorical columns (to be one-hot encoded below)
categorical_cols = dataset[feature_cols].select_dtypes(include=['object']).columns.tolist()
numeric_cols = dataset[feature_cols].select_dtypes(include=[np.number]).columns.tolist()

print(f"Numeric features: {numeric_cols}")
print(f"Categorical features: {categorical_cols}")
print(f"Total observations: {len(dataset):,}")

# Encoding categorical data
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(drop='first'), categorical_cols)], remainder='passthrough')
X = np.array(ct.fit_transform(X))

4.2 Building the Regression Pipeline

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Training the Multiple Linear Regression model on the Training set
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

# Predicting the Test set results
y_pred = regressor.predict(X_test)
np.set_printoptions(precision=2)
print(np.concatenate((y_pred.reshape(len(y_pred),1), y_test.reshape(len(y_test),1)),1))

# Evaluating the model
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print("=" * 50)
print("CLAIM AMOUNT REGRESSION RESULTS")
print("=" * 50)
print(f"RMSE:  ₹{rmse:,.0f}")
print(f"MAE:   ₹{mae:,.0f}")
print(f"R²:    {r2:.3f}")
print(f"  (Context: average claim = ₹{y_test.mean():,.0f})")
print(f"  The prediction error represents {mae/y_test.mean()*100:.1f}% of the average claim")

4.3 Interpreting Coefficients

# Interpreting the coefficients
# After one-hot encoding, the feature names come from the ColumnTransformer
feature_names = ct.get_feature_names_out()

coeff_df = pd.DataFrame({
    'feature': feature_names,
    'coefficient': regressor.coef_
}).sort_values('coefficient', key=abs, ascending=False)

print("Top 10 Most Influential Features (by absolute coefficient):")
print("=" * 60)
print(f"{'Feature':35s} {'Coefficient':>15s} {'Direction':>12s}")
print("-" * 60)
for _, row in coeff_df.head(10).iterrows():
    direction = "Increases claim" if row['coefficient'] > 0 else "Decreases claim"
    print(f"{row['feature']:35s} {row['coefficient']:>+12,.0f}   {direction}")

print(f"\n{'-' * 60}")
print(f"Intercept: {regressor.intercept_:,.0f}")
print(f"\nNote: because the features were standardised (mean 0, std 1), the")
print(f"coefficients are comparable in RELATIVE magnitude (which feature matters")
print(f"most), not in original rupee units. Categorical coefficients are relative")
print(f"to the baseline (dropped) category.")

4.4 Diagnostic Plot

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# LEFT: Actual vs Predicted
axes[0].scatter(y_test, y_pred, alpha=0.3, s=10, color='#6c5ce7')
axes[0].plot([y_test.min(), y_test.max()],
             [y_test.min(), y_test.max()],
             'r--', linewidth=2, label='Perfect Prediction')
axes[0].set_xlabel('Actual Claim Amount')
axes[0].set_ylabel('Predicted Claim Amount')
axes[0].set_title('Regression: Actual vs. Predicted', fontweight='bold')
axes[0].legend()
axes[0].spines['top'].set_visible(False)
axes[0].spines['right'].set_visible(False)

# RIGHT: Residuals
residuals = y_test - y_pred
axes[1].scatter(y_pred, residuals, alpha=0.3, s=10, color='#00d2d3')
axes[1].axhline(y=0, color='red', linestyle='--', linewidth=1.5)
axes[1].set_xlabel('Predicted Claim Amount')
axes[1].set_ylabel('Residual (Actual − Predicted)')
axes[1].set_title('Residual Plot', fontweight='bold')
axes[1].spines['top'].set_visible(False)
axes[1].spines['right'].set_visible(False)

plt.tight_layout()
plt.show()

print("Residual analysis:")
print(f"  Mean residual: {residuals.mean():.0f} (should be close to 0)")
print(f"  Std of residuals: {residuals.std():.0f}")
print(f"  Are residuals roughly evenly distributed above and below zero?")
print(f"  If the residual plot shows a pattern (cone shape, curve), the linear model is inadequate.")

5. Classification — Predicting Claim Probability

Classification predicts a category. In insurance, the most common classification task is predicting whether a policyholder will file a claim (binary: yes or no). This is used for: underwriting (should we accept this risk?), renewal targeting (who is most likely to claim?), and fraud triage (is this claim suspicious?).

5.1 Preparing Data for Classification

# Preparing Data for Classification

# Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset (policy-level: one row per policy, including no-claim policies)
dataset = pd.read_csv('data/insurance_policy_cleaned.csv')

# Check class balance
claim_rate = dataset['has_claim'].mean()
print(f"Claim rate: {claim_rate*100:.1f}%")
print(f"  Claim filed (1): {(dataset['has_claim'] == 1).sum():,}")
print(f"  No claim (0):    {(dataset['has_claim'] == 0).sum():,}")

# Define features and target, and drop rows with missing values
class_features = ['premium', 'age', 'income', 'credit_score', 'sum_assured', 'policy_type']
class_features = [c for c in class_features if c in dataset.columns]
dataset = dataset.dropna(subset=class_features)

X = dataset[class_features]         # keep as DataFrame so ColumnTransformer can use column names
y = dataset['has_claim'].values

# Encoding categorical data
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
categorical_cols = dataset[class_features].select_dtypes(include=['object']).columns.tolist()
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(drop='first'), categorical_cols)], remainder='passthrough')
X = np.array(ct.fit_transform(X))

5.2 Building the Classification Pipeline

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0, stratify = y)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Training the Logistic Regression model on the Training set
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state = 0, class_weight = 'balanced')
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
print(np.concatenate((y_pred.reshape(len(y_pred),1), y_test.reshape(len(y_test),1)),1))

# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
print(cm)
print(accuracy_score(y_test, y_pred))

# Evaluating with precision, recall, and ROC-AUC
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
y_prob = classifier.predict_proba(X_test)[:, 1]
print("=" * 60)
print("CLAIM PROBABILITY CLASSIFICATION RESULTS")
print("=" * 60)
print(f"Accuracy:  {accuracy_score(y_test, y_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall:    {recall_score(y_test, y_pred):.3f}")
print(f"F1 Score:  {f1_score(y_test, y_pred):.3f}")
print(f"ROC-AUC:   {roc_auc_score(y_test, y_prob):.3f}")

5.3 Why Accuracy is Not Enough

In a dataset where only 12% of policyholders file a claim, a naive model that predicts "No Claim" for everyone achieves 88% accuracy — but it is completely useless. This is the class imbalance problem that pervades insurance ML. Accuracy alone is dangerously misleading.

For insurance classification problems, the evaluation metric depends on the business context:

• Fraud detection: Recall (catching fraud) matters more than precision (false accusations are expensive but catching fraud saves money). But if precision is too low, investigators waste time on false positives.
• Claims triage: Precision on high-severity claims is critical — you do not want to misclassify a catastrophic claim as low-severity. But recall on fast-track claims must be high enough to achieve efficiency gains.
• Underwriting: Both precision and recall matter because rejecting a good applicant loses revenue and accepting a bad applicant generates losses. The threshold must be calibrated to the insurer's risk appetite.
• ROC-AUC: This metric measures the model's ability to rank risks correctly — how well it separates the claim-filers from the non-claim-filers. A model with ROC-AUC of 0.85 means that 85% of the time, a randomly chosen claim-filer will have a higher predicted probability than a randomly chosen non-claim-filer. This is the most commonly reported classification metric in insurance ML.

5.4 ROC Curve

# Plot ROC curve
from sklearn.metrics import roc_curve, roc_auc_score
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
auc = roc_auc_score(y_test, y_prob)

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(fpr, tpr, color='#6c5ce7', linewidth=2.5,
        label=f'ROC Curve (AUC = {auc:.3f})')
ax.plot([0, 1], [0, 1], 'r--', linewidth=1.5, label='Random Classifier (AUC = 0.5)')
ax.fill_between(fpr, tpr, alpha=0.15, color='#6c5ce7')
ax.set_xlabel('False Positive Rate (1 − Specificity)')
ax.set_ylabel('True Positive Rate (Recall)')
ax.set_title('ROC Curve — Claim Prediction Model', fontweight='bold')
ax.legend(loc='lower right')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

# Find the optimal threshold (Youden's J statistic)
j_scores = tpr - fpr
best_idx = np.argmax(j_scores)
best_threshold = thresholds[best_idx]
ax.annotate(f'Optimal threshold: {best_threshold:.2f}',
            xy=(fpr[best_idx], tpr[best_idx]),
            xytext=(fpr[best_idx] + 0.15, tpr[best_idx] - 0.1),
            arrowprops=dict(arrowstyle='->', color='green'),
            fontsize=10, color='green')
ax.plot(fpr[best_idx], tpr[best_idx], 'go', markersize=10)

plt.tight_layout()
plt.show()

print(f"\nOptimal threshold (Youden's J): {best_threshold:.3f}")
print(f"  At this threshold: Sensitivity = {tpr[best_idx]:.3f}, Specificity = {1-fpr[best_idx]:.3f}")
print(f"\nInterpretation: A model with AUC = {auc:.3f} can distinguish")
print(f"claim-filers from non-claim-filers {auc*100:.1f}% of the time.")
print(f"This is {'strong' if auc > 0.85 else 'moderate' if auc > 0.7 else 'weak'} predictive power.")

6. Clustering — Segmenting Policyholders

Clustering is an unsupervised learning technique that groups similar data points together. In insurance, clustering is used for customer segmentation, risk profiling, fraud detection (unusual clusters), and product design (identifying underserved segments). Unlike regression and classification, clustering does not use a target variable — it finds structure in the data purely from the features.

6.1 When to Cluster

Use clustering when you want to discover groups in your data that you did not know existed. Common insurance applications:

• Customer segmentation: Group policyholders by demographics, policy behaviour, and claims history to create differentiated service, pricing, and marketing strategies.
• Risk profiling: Identify natural risk groups that may not align with traditional rating factors — revealing overpriced and underpriced segments.
• Fraud pattern discovery: Detect clusters of claims with unusual feature combinations that may indicate organised fraud rings.
• Product design: Identify underserved customer groups whose needs are not met by existing products.

6.2 K-Means Clustering in Python

# K-Means Clustering

# Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset (policy-level)
dataset = pd.read_csv('data/insurance_policy_cleaned.csv')

# Building a customer-level profile (one row per customer, including no-claim customers)
customer_profile = dataset.groupby('customer_id').agg(
    age = ('age', 'first'),
    income = ('income', 'first'),
    credit_score = ('credit_score', 'first'),
    total_premium = ('premium', 'sum'),
    total_claims = ('total_claim_amount', 'sum'),
    claim_count = ('n_claims', 'sum'),
    policy_count = ('policy_id', 'nunique')
).reset_index()

# Adding derived features
customer_profile['loss_ratio'] = (customer_profile['total_claims'] / customer_profile['total_premium']).fillna(0)
customer_profile['claims_per_policy'] = customer_profile['claim_count'] / customer_profile['policy_count']

# Selecting the features for clustering, and dropping rows with missing values
clust_cols = ['age', 'income', 'credit_score', 'total_premium', 'loss_ratio', 'claims_per_policy', 'policy_count']
clust_cols = [c for c in clust_cols if c in customer_profile.columns]
customer_profile = customer_profile.dropna(subset=clust_cols)
X = customer_profile[clust_cols].values

# Feature Scaling (K-means is distance-based, so scaling is critical)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit_transform(X)

# Using the elbow method to find the optimal number of clusters
from sklearn.cluster import KMeans
wcss = []
for i in range(1, 11):
    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 42)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)
plt.plot(range(1, 11), wcss)
plt.title('The Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()

6.3 Profiling the Clusters

# Training the K-Means model on the dataset
kmeans = KMeans(n_clusters = 4, init = 'k-means++', random_state = 42)
y_kmeans = kmeans.fit_predict(X)

# Profiling the clusters
customer_profile['cluster'] = y_kmeans
cluster_profile = customer_profile.groupby('cluster').agg(
    count = ('customer_id', 'count'),
    avg_age = ('age', 'mean'),
    avg_income = ('income', 'mean'),
    avg_credit = ('credit_score', 'mean'),
    avg_premium = ('total_premium', 'mean'),
    avg_loss_ratio = ('loss_ratio', 'mean'),
    avg_claims_per_policy = ('claims_per_policy', 'mean'),
    avg_policies = ('policy_count', 'mean')
).round(1)

# Sort by cluster size for readability
cluster_profile = cluster_profile.sort_values('count', ascending=False)

print("=" * 90)
print("CUSTOMER SEGMENTATION — CLUSTER PROFILES")
print("=" * 90)
print(cluster_profile.to_string())

# Give each cluster a meaningful name based on its profile
print(f"\n{'='*90}")
print("BUSINESS INTERPRETATION:")
print(f"{'='*90}")
for cluster_id in cluster_profile.index:
    row = cluster_profile.loc[cluster_id]
    share = row['count'] / cluster_profile['count'].sum() * 100
    # Generate a descriptive label
    if row['avg_loss_ratio'] < 0.3 and row['avg_income'] > cluster_profile['avg_income'].median():
        label = "Low-Risk, High-Value"
    elif row['avg_loss_ratio'] > 0.6:
        label = "High-Risk — Manage Carefully"
    elif row['avg_claims_per_policy'] < 0.1 and row['avg_policies'] > 2:
        label = "Loyal, Low-Claim — ⭐ Protect"
    elif row['avg_age'] > 50:
        label = "Senior Segment — Growing"
    else:
        label = "Standard Risk — Optimize Service"
    print(f"  Cluster {cluster_id} ({share:.0f}% of customers): {label}")
    print(f"    Age: {row['avg_age']:.0f}, Income: ₹{row['avg_income']:,.0f}, Credit: {row['avg_credit']:.0f}")
    print(f"    Loss Ratio: {row['avg_loss_ratio']*100:.1f}%, Policies: {row['avg_policies']:.1f}, Claims/Policy: {row['avg_claims_per_policy']:.2f}")
    print()

7. Overfitting, Underfitting, and Cross-Validation

A model that memorizes the training data but fails on new data is overfitted. A model that is too simple to capture the patterns in the data is underfitted. Managing this tradeoff is the central challenge of applied machine learning — and it is especially acute in insurance, where the cost of a wrong prediction (an underpriced risk, a missed fraud) can be measured in crores.

7.1 Overfitting vs. Underfitting

7.2 Detecting Overfitting

# Detecting Overfitting (continues from the regression model in Section 4)

# Re-load and re-prepare the regression data so this section is self-contained
# (Sections 5-6 redefined X and y, so re-establish the regression context here).
import numpy as np
import pandas as pd
dataset = pd.read_csv('data/insurance_cleaned.csv')
dataset = dataset[dataset['claim_amount'].notna()].copy()
feature_cols = ['age', 'income', 'credit_score', 'premium', 'sum_assured', 'policy_type']
feature_cols = [c for c in feature_cols if c in dataset.columns]
dataset = dataset.dropna(subset=feature_cols)
X = dataset[feature_cols]          # keep as DataFrame so ColumnTransformer can use column names
y = dataset['claim_amount'].values
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(drop='first'), ['policy_type'])], remainder='passthrough')
X = np.array(ct.fit_transform(X))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

# Training a deliberately over-complex model (deep decision tree) on the Training set
from sklearn.tree import DecisionTreeRegressor
overfit_tree = DecisionTreeRegressor(max_depth = None, min_samples_leaf = 1, random_state = 0)
overfit_tree.fit(X_train, y_train)

# Evaluating the overfit model
train_score = overfit_tree.score(X_train, y_train)
test_score = overfit_tree.score(X_test, y_test)

print("OVERFITTING DEMONSTRATION — Decision Tree (no depth limit)")
print("=" * 55)
print(f"  Training R²:  {train_score:.4f}  {'(unrealistically high — memorized training data)' if train_score > 0.95 else ''}")
print(f"  Test R²:      {test_score:.4f}")
print(f"  Gap:          {train_score - test_score:.4f}")
print()
if train_score > 0.95:
    print("  ⚠ This model has memorized the training data. It will perform")
    print("    poorly on new policyholders. The training R² is misleadingly high.")
    print()
    print("  Solution: Limit tree depth, increase min_samples_leaf, or use cross-validation.")
else:
    print("  The model shows modest overfitting. Consider regularization.")

# Training a regularized model (limited depth) to avoid overfitting
regularized_tree = DecisionTreeRegressor(max_depth = 5, min_samples_leaf = 20, random_state = 0)
regularized_tree.fit(X_train, y_train)
train_score_r = regularized_tree.score(X_train, y_train)
test_score_r = regularized_tree.score(X_test, y_test)

print(f"\nREGULARIZED DECISION TREE (max_depth=5, min_samples_leaf=20)")
print(f"  Training R²:  {train_score_r:.4f}")
print(f"  Test R²:      {test_score_r:.4f}")
print(f"  Gap:          {train_score_r - test_score_r:.4f}")
print(f"\nThe regularized model has a SMALLER gap between train and test,")
print(f"which means it will generalize better to new data.")

7.3 Cross-Validation

Cross-validation evaluates model performance on multiple train/test splits, providing a more reliable estimate of how the model will perform on new data. It reduces the variance of the performance estimate — a single train/test split may be lucky or unlucky depending on which observations ended up in the test set.

# Cross-Validation

from sklearn.model_selection import cross_val_score, KFold

# Set up 5-fold cross-validation
cv = KFold(n_splits = 5, shuffle = True, random_state = 0)

# Evaluate the regression model with cross-validation
cv_scores = cross_val_score(regressor, X, y, cv = cv, scoring = 'r2')

print("CROSS-VALIDATION RESULTS (5-Fold)")
print("=" * 40)
print(f"  Fold 1 R²: {cv_scores[0]:.3f}")
print(f"  Fold 2 R²: {cv_scores[1]:.3f}")
print(f"  Fold 3 R²: {cv_scores[2]:.3f}")
print(f"  Fold 4 R²: {cv_scores[3]:.3f}")
print(f"  Fold 5 R²: {cv_scores[4]:.3f}")
print(f"  ─────────────────────────")
print(f"  Mean R²:   {cv_scores.mean():.3f}")
print(f"  Std Dev:   {cv_scores.std():.3f}")
print(f"\nIf the standard deviation is large (>0.1), the model's performance")
print(f"varies significantly across data splits — investigate stability.")
print(f"If mean R² is close to the single-test R² from our 80/20 split,")
print(f"we can be more confident in the performance estimate.")

References

1. Gartner, "Data and Analytics Maturity Score," Gartner, Inc. [Online]. Available: https://www.gartner.com/en/data-analytics/research/data-analytics-maturity-score
2. McKinsey & Company, "On the brink: Realizing the value of analytics in insurance," McKinsey & Company. [Online]. Available: https://www.mckinsey.com/industries/financial-services/our-insights/on-the-brink-realizing-the-value-of-analytics-in-insurance
3. Deloitte, "Digital Insurance Maturity 2025," Deloitte. [Online]. Available: https://www.deloitte.com/ce/en/related-content/digital-insurance-maturity-2025.html
4. McKinsey & Company, "The future of AI in the insurance industry," 2025. [Online]. Available: https://www.mckinsey.com/industries/financial-services/our-insights/the-future-of-ai-in-the-insurance-industry

Further reading (Task G5): the applied-research anchor for this session's finance framing is McKinsey [4]. Flag for instructor review: a peer-reviewed Journal of Risk and Insurance / Geneva Papers article on analytics-maturity or ML-adoption business value was not verified during this pass and should be added before publication.