Session 12: AI for Underwriting

1. Underwriting: From Art to Science

Underwriting is the process of evaluating risk and deciding whether to accept it, at what price, and on what terms. For most of insurance history, underwriting was an art — a senior underwriter in a room, reviewing paper applications, applying experience-based rules, and making subjective judgments. The problem was not that this was inefficient (though it was). The problem was that it was inconsistent — two equally qualified underwriters reviewing the same application could reach different decisions, and the same underwriter could make different decisions at 9 AM and 4 PM.

AI-driven underwriting replaces subjective judgment with consistent, data-driven risk assessment. It does not eliminate the underwriter — it eliminates the inconsistency. The underwriter's role shifts from making individual risk decisions to: (a) setting the risk appetite and pricing framework within which the AI operates, (b) handling the complex referrals the AI flags, (c) monitoring the AI's decisions for drift and fairness, and (d) improving the models over time.

1.1 Traditional vs. AI Underwriting

2. Feature Engineering for Underwriting

Feature engineering is the process of transforming raw data into inputs that a machine learning model can use effectively. In underwriting, this means taking the application data, internal records, and external data sources and creating predictive features. The quality of feature engineering determines more about model performance than the choice of algorithm — a well-engineered dataset with a simple model often outperforms a poorly-engineered dataset with a complex one.

2.1 Feature Types in Underwriting

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, classification_report, confusion_matrix
import matplotlib.pyplot as plt

# Load the insurance dataset
df = pd.read_csv('data/insurance_cleaned.csv')

# Create policy-level dataset (one row per policy)
dataset = df.groupby('policy_id').agg(
    customer_id=('customer_id', 'first'),
    age=('age', 'first'),
    gender=('gender', 'first'),
    location=('location', 'first'),
    occupation=('occupation', 'first'),
    income=('income', 'first'),
    credit_score=('credit_score', 'first'),
    premium=('premium', 'first'),
    sum_assured=('sum_assured', 'first'),
    policy_type=('policy_type', 'first'),
    channel=('channel', 'first'),
    tenure_months=('tenure_months', 'first'),
    has_claim=('claim_id', lambda x: 1 if len(x) > 0 else 0),
    claim_amount=('claim_amount', 'sum'),
    days_since_purchase=('start_date', lambda x:
        (pd.Timestamp.today() - pd.to_datetime(x.iloc[0])).days if len(x) > 0 else 0)
).reset_index()

print(f"Policy-level dataset: {dataset.shape[0]:,} rows, {dataset.shape[1]:,} columns")
print(f"\nClaim rate: {dataset['has_claim'].mean()*100:.1f}%")
print(f"Columns available for feature engineering:\n{dataset.columns.tolist()}")

2.2 Creating Underwriting Features

# === FEATURE ENGINEERING PIPELINE ===

# 1. Risk-to-Income Ratio (captures affordability stress)
dataset['risk_income_ratio'] = (
    dataset['sum_assured'] / (dataset['income'] + 1)
)
# Cap extreme values
dataset['risk_income_ratio'] = dataset['risk_income_ratio'].clip(upper=100)

# 2. Premium Density (premium per unit of sum assured)
dataset['premium_density'] = (
    dataset['premium'] / (dataset['sum_assured'] + 1)
)

# 3. Age squared (non-linear age effect — claim risk often increases faster than linearly)
dataset['age_squared'] = dataset['age'] ** 2

# 4. Credit score category (binned version)
dataset['credit_tier'] = pd.cut(
    dataset['credit_score'],
    bins=[0, 600, 700, 800, 900],
    labels=['Low (<600)', 'Fair (600-699)', 'Good (700-799)', 'Excellent (800+)']
)

# 5. Policy tenure in years (from tenure_months)
dataset['tenure_years'] = dataset['tenure_months'] / 12

# 6. Age × Channel interaction (certain channels attract different risk profiles)
dataset['age_channel'] = dataset['age'] * (
    dataset['channel'].map({'Online': 1, 'Agent': 2, 'Aggregator': 3, 'Bancassurance': 4, 'Partnership': 5})
    .fillna(0)
)

# 7. Income decile (within the portfolio)
dataset['income_decile'] = pd.qcut(
    dataset['income'], q=10, labels=False, duplicates='drop'
)

# 8. Senior citizen flag (biologically significant risk factor)
dataset['senior_flag'] = (dataset['age'] >= 60).astype(int)

# 9. Low credit flag
dataset['low_credit_flag'] = (dataset['credit_score'] < 650).astype(int)

# Verify new features
new_features = ['risk_income_ratio', 'premium_density', 'age_squared',
                'credit_tier', 'tenure_years', 'age_channel',
                'income_decile', 'senior_flag', 'low_credit_flag']

print("New features created:")
for feat in new_features:
    if feat in dataset.columns:
        print(f"  ✓ {feat:25s} — dtype: {str(dataset[feat].dtype):10s}, "
              f"unique: {dataset[feat].nunique()}, missing: {dataset[feat].isna().sum()}")
    else:
        print(f"  ✗ {feat:25s} — NOT FOUND")

2.3 Feature Selection

Not all features are worth including. Features with: (a) too many missing values, (b) near-zero variance (almost same value for every observation), or (c) correlations > 0.9 with another feature (multicollinearity) should be reviewed before modeling. Feature selection balances predictive power against model complexity and explainability.

# Check for highly correlated features
numeric_features_for_uw = ['age', 'income', 'credit_score', 'premium',
                           'sum_assured', 'tenure_months', 'risk_income_ratio',
                           'premium_density', 'age_squared', 'tenure_years',
                           'age_channel', 'income_decile', 'senior_flag',
                           'low_credit_flag']
numeric_features_for_uw = [c for c in numeric_features_for_uw if c in dataset.columns]

corr_matrix = dataset[numeric_features_for_uw].corr()

# Find high-correlation pairs
high_corr_pairs = []
for i in range(len(corr_matrix.columns)):
    for j in range(i + 1, len(corr_matrix.columns)):
        if abs(corr_matrix.iloc[i, j]) > 0.85:
            high_corr_pairs.append((corr_matrix.columns[i], corr_matrix.columns[j],
                                    corr_matrix.iloc[i, j]))

if high_corr_pairs:
    print("⚠ HIGH CORRELATION DETECTED (|r| > 0.85) — consider removing one of each pair:")
    for v1, v2, corr in high_corr_pairs:
        print(f"  {v1:20s} ↔ {v2:20s}  r = {corr:.3f}")
else:
    print("✓ No highly correlated feature pairs detected")

# Separate features and target
feature_cols = (numeric_features_for_uw +
                ['policy_type', 'channel', 'gender', 'occupation', 'credit_tier'])
feature_cols = [c for c in feature_cols if c in dataset.columns]

X = dataset[feature_cols].copy()
y = dataset['has_claim']

print(f"\nFinal feature set: {X.shape[1]} features ({X.shape[0]:,} observations)")
print(f"Categorical: {X.select_dtypes(include=['object', 'category']).columns.tolist()}")
print(f"Numerical: {X.select_dtypes(include=[np.number]).columns.tolist()}")

3. Building a Risk Scoring Model

A risk score is a single number (typically 0–1000) that represents the estimated risk of a policyholder filing a claim. The score is derived from the predicted probability from a classification model, scaled to a more interpretable range. Higher score = higher risk. The business use: a risk score of 200 means "low risk" (accept and possibly discount), 500 means "standard risk" (accept at standard terms), and 800 means "high risk" (load premium, add restrictions, or refer).

3.1 Training a Random Forest Classifier

# Identify categorical and numeric columns
cat_cols = X.select_dtypes(include=['object', 'category']).columns.tolist()
num_cols = X.select_dtypes(include=[np.number]).columns.tolist()

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0, stratify = y)

# Encoding categorical data
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(drop='first', handle_unknown='ignore'), cat_cols)], remainder='passthrough')
X_train = np.array(ct.fit_transform(X_train))
X_test = np.array(ct.transform(X_test))

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Training the Random Forest model on the Training set
from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators = 200, max_depth = 10, min_samples_leaf = 20, class_weight = 'balanced', random_state = 0, n_jobs = -1)
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_prob = classifier.predict_proba(X_test)[:, 1]

# Evaluating the model
from sklearn.metrics import roc_auc_score, confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
print(cm)
print(accuracy_score(y_test, y_pred))
auc = roc_auc_score(y_test, y_prob)
print(f"Model ROC-AUC: {auc:.3f}")
print(f"Interpretation: The model can distinguish claim-filers from non-claim-filers")
print(f"{auc*100:.1f}% of the time — this is {'strong' if auc > 0.8 else 'moderate' if auc > 0.7 else 'moderate-to-weak'} discriminatory power.")

3.2 Converting Probabilities to Risk Scores

# Predict probabilities for ALL policies (not just test set)
X_all = sc.transform(np.array(ct.transform(X)))
all_prob = classifier.predict_proba(X_all)[:, 1]

# Scale to 0–1000 risk score
dataset['risk_score'] = (all_prob * 1000).astype(int)

# Define risk tiers
dataset['risk_tier'] = pd.cut(
    dataset['risk_score'],
    bins=[0, 200, 400, 600, 800, 1000],
    labels=['Very Low', 'Low', 'Moderate', 'High', 'Very High'],
    right=False
)

# Examine risk score distribution
print("Risk Score Distribution:")
print(f"  Min:      {dataset['risk_score'].min()}")
print(f"  25th pct: {dataset['risk_score'].quantile(0.25):.0f}")
print(f"  Median:   {dataset['risk_score'].median():.0f}")
print(f"  75th pct: {dataset['risk_score'].quantile(0.75):.0f}")
print(f"  Max:      {dataset['risk_score'].max()}")
print(f"\nRisk Tier Breakdown:")
print(dataset['risk_tier'].value_counts().sort_index())

# Validate risk tiers against actual claim rates
tier_validation = dataset.groupby('risk_tier', observed=False).agg(
    count=('risk_tier', 'size'),
    actual_claim_rate=('has_claim', 'mean'),
    avg_risk_score=('risk_score', 'mean')
).round(3)

print(f"\n{'=' * 65}")
print(f"{'Risk Tier':15s} {'Count':>8s} {'%':>5s} {'Claim Rate':>12s} {'Avg Score':>10s}")
print(f"{'=' * 65}")
for tier in tier_validation.index:
    r = tier_validation.loc[tier]
    print(f"{str(tier):15s} {r['count']:>5,.0f}  {r['count']/tier_validation['count'].sum()*100:>4.1f}%    {r['actual_claim_rate']*100:>5.1f}%     {r['avg_risk_score']:>5.0f}")

3.3 The Calibration Check

A well-calibrated risk scoring model should satisfy: for policies with a risk score of X, approximately X/1000 of them should actually file a claim. For example, policies with a risk score of 500 should have an actual claim rate of ~50%. If the model says 500 but the actual rate is 35%, the model is miscalibrated — it is overstating risk. If it says 500 and the actual rate is 65%, it is understating risk.

# Calibration check: compare predicted risk score to actual claim rate by decile
dataset['score_decile'] = pd.qcut(dataset['risk_score'], q=10, labels=False, duplicates='drop')

calibration = dataset.groupby('score_decile').agg(
    avg_predicted_risk=('risk_score', 'mean'),
    actual_claim_rate=('has_claim', 'mean'),
    count=('risk_score', 'count')
)

# The predicted risk / 1000 should approximate the actual claim rate
calibration['expected_claim_rate'] = calibration['avg_predicted_risk'] / 1000
calibration['calibration_error'] = (
    calibration['actual_claim_rate'] - calibration['expected_claim_rate']
)

print(f"{'Decile':8s} {'Predicted':>10s} {'Expected':>10s} {'Actual':>10s} {'Error':>10s}")
print("-" * 48)
for idx, row in calibration.iterrows():
    print(f"{idx:>3d}-{idx+1:.0f}%   {row['avg_predicted_risk']:>5.0f}     {row['expected_claim_rate']*100:>5.1f}%     {row['actual_claim_rate']*100:>5.1f}%    {row['calibration_error']*100:>+5.1f}%")

print(f"\nMean absolute calibration error: {calibration['calibration_error'].abs().mean()*100:.2f}%")
print("(Lower is better. <3% mean error is acceptable for production models.)")

4. Random Forest & Tree-Based Models

Random Forest is the most widely used algorithm for insurance underwriting models. It is popular because it: (a) handles mixed data types (numeric and categorical) naturally, (b) captures non-linear relationships without manual feature transformation, (c) provides built-in feature importance rankings, (d) is less prone to overfitting than single decision trees, and (e) is relatively robust to outliers and missing data.

4.1 How Random Forest Works

A Random Forest builds hundreds of individual decision trees, each trained on a random subset of the data and a random subset of features. Each tree independently predicts the outcome (claim or no claim). The forest's final prediction is the average of all individual tree predictions. This "ensemble" approach corrects for the tendency of individual trees to overfit — the trees make different mistakes, and the averaging cancels those mistakes out.

4.2 Hyperparameter Tuning

# Key Random Forest hyperparameters and their effect on underwriting models
print("RANDOM FOREST HYPERPARAMETERS FOR UNDERWRITING")
print("=" * 60)
print(f"{'Parameter':25s} {'Controls':40s}")
print("-" * 60)
print(f"{'n_estimators':25s} Number of trees. More trees = more stable predictions.")
print(f"{'':25s}     Diminishing returns beyond 300–500 trees. Set high and use n_jobs.")
print(f"{'max_depth':25s} Maximum tree depth. Deeper trees = more complex (risk of overfitting).")
print(f"{'':25s}     For underwriting, max_depth=10–15 works well with 100K+ records.")
print(f"{'min_samples_leaf':25s} Minimum observations per leaf. Higher = simpler, smoother predictions.")
print(f"{'':25s}     20–50 for large datasets; prevents the model from learning noise.")
print(f"{'class_weight':25s} Handles class imbalance. Set to 'balanced' or 'balanced_subsample'.")
print(f"{'':25s}     Tells the model: "false negatives are more costly than false positives."")
print(f"{'max_features':25s} Features considered per split. sqrt(n) for classification is standard.")
print(f"{'':25s}     Lower = more randomness = less correlated trees = better ensemble.")

# Quick grid search for optimal depth
from sklearn.model_selection import cross_val_score

depths = [5, 8, 10, 12, 15]
cv_scores_depth = []

for depth in depths:
    rf = RandomForestClassifier(
        n_estimators=100, max_depth=depth, min_samples_leaf=20,
        class_weight='balanced', random_state = 0, n_jobs=-1
    )
    pipe = Pipeline(steps=[
        ('preprocessor', ct),
        ('classifier', rf)
    ])
    scores = cross_val_score(pipe, X_train, y_train, cv=3, scoring='roc_auc')
    cv_scores_depth.append(scores.mean())
    print(f"  max_depth={depth:>2d}: CV AUC = {scores.mean():.4f}")

best_depth = depths[np.argmax(cv_scores_depth)]
print(f"\nBest max_depth: {best_depth}")

4.3 Feature Importance

# Extract feature importance from the trained model
classifier = classifier

# Get feature names after preprocessing
preprocessor_trained = ct
cat_encoder = preprocessor_trained.named_transformers_['cat']

# Get category names
cat_feature_names = []
if len(cat_cols) > 0:
    cat_feature_names = cat_encoder.get_feature_names_out(cat_cols).tolist()
all_features = num_cols + cat_feature_names

# Get importances
importances = classifier.feature_importances_

# Create DataFrame
fi_df = pd.DataFrame({
    'feature': all_features,
    'importance': importances
}).sort_values('importance', ascending=True)

# Plot
fig, ax = plt.subplots(figsize=(10, max(6, len(fi_df) * 0.3)))
ax.barh(fi_df.tail(15)['feature'], fi_df.tail(15)['importance'],
        color='#6c5ce7', edgecolor='white')
ax.set_xlabel('Feature Importance (Random Forest)')
ax.set_title('Top 15 Features Driving Underwriting Risk Score', fontweight='bold')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
plt.tight_layout()
plt.show()

print("\nTop 5 Risk Drivers:")
for _, row in fi_df.sort_values('importance', ascending=False).head(5).iterrows():
    print(f"  {row['feature']:25s}: {row['importance']:.4f} ({row['importance']/fi_df['importance'].sum()*100:.1f}% of total importance)")

5. Premium Recommendation Engine

The business value of an underwriting model is not the risk score itself — it is the decision that the risk score enables. The premium recommendation engine maps risk scores to pricing actions: what premium should we charge this specific policyholder given their risk score, compared to the current "standard" premium?

5.1 Mapping Risk Scores to Premium Adjustments

# Define premium adjustment rules based on risk tier
# The "standard premium" is the base rate for a Moderate risk policyholder
# Very Low risk:  20–30% discount
# Low risk:       5–15% discount
# Moderate risk:  Standard rate (±5% from base)
# High risk:      10–25% loading
# Very High risk: 25–50% loading OR referral for manual review

def calculate_premium_loading(risk_score, base_premium):
    """Calculate recommended premium adjustment based on risk score.

    Parameters:
    -----------
    risk_score : int — Model risk score (0–1000)
    base_premium : float — The current standard premium for this product

    Returns:
    --------
    dict — Loading factor, recommended premium, and action
    """
    if risk_score < 200:
        factor = 0.70  # 30% discount
        action = "Accept — Fast-track (no human review)"
    elif risk_score < 400:
        factor = 0.85  # 15% discount
        action = "Accept — Standard review"
    elif risk_score < 600:
        factor = 1.00  # Standard rate
        action = "Accept — Standard review"
    elif risk_score < 750:
        factor = 1.20  # 20% loading
        action = "Accept with loading — Manual review recommended"
    elif risk_score < 900:
        factor = 1.40  # 40% loading
        action = "Refer — Senior underwriter decision required"
    else:
        factor = None  # Decline / manual referral
        action = "Decline — Or refer for exceptional approval"

    if factor is not None:
        recommended_premium = round(base_premium * factor, 0)
    else:
        recommended_premium = None

    return {
        'risk_score': risk_score,
        'base_premium': base_premium,
        'loading_factor': factor,
        'recommended_premium': recommended_premium,
        'action': action,
        'premium_change_pct': round((factor - 1) * 100, 0) if factor else None
    }

# Apply to sample policies
sample_policies = dataset[['policy_id', 'risk_score', 'premium', 'risk_tier']].sample(10, random_state = 0)

print("=" * 120)
print(f"{'Policy ID':12s} {'Risk Score':>12s} {'Tier':15s} {'Base Premium':>15s} {'Loading':>8s} {'Recommended':>15s} {'Action'}")
print("=" * 120)
for _, row in sample_policies.sort_values('risk_score').iterrows():
    result = calculate_premium_loading(row['risk_score'], row['premium'])
    loading_str = f"{result['premium_change_pct']:+.0f}%" if result['loading_factor'] else "N/A"
    rec_str = f"₹{result['recommended_premium']:,.0f}" if result['recommended_premium'] else "DECLINE"
    print(f"{row['policy_id']:12s} {result['risk_score']:>5d}      {str(row['risk_tier']):15s} ₹{row['premium']:>8,.0f}          {loading_str:>6s}  {rec_str:>12s}  {result['action'][:40]}")

# Portfolio impact analysis
dataset['premium_loading_factor'] = dataset['risk_score'].apply(
    lambda s: 0.70 if s < 200 else 0.85 if s < 400 else 1.00 if s < 600
    else 1.20 if s < 750 else 1.40 if s < 900 else None
)
dataset['recommended_premium'] = dataset['premium'] * dataset['premium_loading_factor']

print(f"\n{'=' * 60}")
print("PORTFOLIO IMPACT SUMMARY")
print(f"{'=' * 60}")
print(f"{'Tier':15s} {'Policies':>10s} {'Current Premium':>18s} {'New Premium':>15s} {'Change':>10s}")
print("-" * 68)
for tier in ['Very Low', 'Low', 'Moderate', 'High', 'Very High']:
    tier_data = dataset[dataset['risk_tier'] == tier]
    if len(tier_data) == 0:
        continue
    current = tier_data['premium'].sum()
    new = tier_data['recommended_premium'].sum()
    change = new - current
    print(f"{tier:15s} {len(tier_data):>6,d}     ₹{current/1e7:>6.1f}Cr        ₹{new/1e7:>6.1f}Cr     {'+' if change > 0 else ''}₹{change/1e7:>4.1f}Cr")

total_current = dataset['premium'].sum()
total_new = dataset['recommended_premium'].sum() if dataset['recommended_premium'].notna().all() else None
print("-" * 68)
if total_new:
    print(f"{'TOTAL':15s} {len(dataset):>6,d}     ₹{total_current/1e7:>6.1f}Cr        ₹{total_new/1e7:>6.1f}Cr     {'+' if total_new > total_current else ''}₹{(total_new-total_current)/1e7:>4.1f}Cr")

References

1. National Association of Insurance Commissioners (NAIC), "Model Bulletin: Use of Artificial Intelligence Systems by Insurers," NAIC, Kansas City, MO, USA, 2023. [Online]. Available: https://content.naic.org/sites/default/files/call_materials/REGULATORY%20GUIDANCE%20PCKG%206-3-24.pdf
2. MDPI Risks, "Algorithmic Bias Under the EU AI Act: Compliance Risk, Capital Strain, and Pricing Distortions in Life and Health Insurance Underwriting," Risks, vol. 13, no. 9, art. 160, 2025. [Online]. Available: https://www.mdpi.com/2227-9091/13/9/160
3. Insurance Business Magazine (Asia), "India's insurance regulator steps in to govern AI adoption," June 2026. [Online]. Available: https://www.insurancebusinessmag.com/asia/news/technology/indias-insurance-regulator-steps-in-to-govern-ai-adoption-579846.aspx

Required reading (Task G5): MDPI Risks [2] is the finance-anchor academic reading for this session — it connects underwriting-model bias to capital strain and pricing distortion under the EU AI Act, and is peer-reviewed (Risks is an MDPI journal).