Build a calibrated financial risk scoring engine

4 of 4 applicants correctly classified at this λ

Four applicants, three risk classes, one softmax classifier. At λ=0 every prediction matches the true risk. Push λ up and watch predictions start flipping to the wrong class — while the probability bars stay just as confident-looking as before.

The engine is multinomial logistic regression — softmax over a per-class linear score — with the same weight-shrinkage mechanism from the ridge chapter applied to every class's weights at once:

• $\mathbf{w}_k^{\text{raw}}$ — class $k$'s unregularized weight vector, over the applicant's features.
• $b_k$ — class $k$'s bias — left unpenalized, exactly as in the ridge and lasso chapters.
• $\lambda$ — the shrinkage strength; this $1/(1+\lambda)$ form is ridge's own closed-form shrinkage in the special case of standardized, uncorrelated features.
• $p_k(\mathbf{x})$ — the calibrated probability applicant $\mathbf{x}$ belongs to risk class $k$.

1. Regularization here isn't about sparsity — it's about calibration

   Unlike lasso, nothing here is meant to zero out; the goal is to keep any one class's weights from dominating the score purely due to unconstrained magnitude.

2. Softmax doesn't know the weights were regularized

   Softmax just normalizes whatever logits it's handed — it has no way to flag "these logits came from an over-shrunk model" versus "these logits are trustworthy." A miscalibrated model can look exactly as confident as a good one.

3. Over-regularization has a real direction of failure here

   As every class's weights shrink toward zero, the classifier increasingly falls back on each class's bias alone — and the bias ranking (low > medium > high) means over-shrinking systematically under-estimates risk, not randomly misclassifies it.

accuracy: 1/4 — average confidence in the winning class: 78.1%

Watch accuracy and average confidence side by side as λ moves. Accuracy falls off a cliff between λ=0.2 and λ=0.3 — but average confidence barely moves, staying in the 70s-to-80s percent the entire time. Confidence alone would never warn you that the model just started getting people wrong.

Bao: debt ratio 5, 0 late payments — a lender's judgment: medium risk.

1. At λ=0.2, correctly medium — but barely

   Shrunk weights give logits (low, medium, high) = $(1.833, 2.000, 0.167)$. Exponentiate each:

   • $e^{1.833} \approx 6.252$
   • $e^{2.000} \approx 7.389$
   • $e^{0.167} \approx 1.182$

   Dividing by the sum $6.252+7.389+1.182 \approx 14.823$: $p \approx (0.422, 0.498, 0.080)$ — medium wins, but by a margin of less than 0.08.

2. At λ=0.3, that thin margin flips

   Logits become $(2.154, 2.000, -0.154)$ — low has overtaken medium. Exponentiate each:

   • $e^{2.154} \approx 8.621$
   • $e^{2.000} \approx 7.389$
   • $e^{-0.154} \approx 0.857$

   Dividing by the sum $8.621+7.389+0.857 \approx 16.867$: $p \approx (0.511, 0.438, 0.051)$. Bao is now predicted low risk: an under-flagged applicant, purely from one notch more shrinkage.

3. Confidence gives no warning either way

   Bao's top-class probability is $0.498$ right before the flip and $0.511$ right after — essentially unchanged. The model is equally "sure" of two opposite answers.