Part IV — Supervised Learning: Regression & Linear Classifiers · Chapter 8

Multinomial logistic regression & softmax

Hook

Sigmoid squashes one number into a probability between 0 and 1 — perfect for two classes, "yes" or "no." But what happens when there are three classes, or ten? You need something that turns a whole vector of scores into a whole vector of probabilities, one per class, all landing between 0 and 1 and summing to exactly 1.

Intuition

Three raw scores ("logits"), one per class. Drag any slider and every probability updates — raise one class's logit and the others don't just drop independently, they get squeezed out proportionally, because the three numbers are locked together by that "sums to 1" constraint.

Formalize

Softmax turns a vector of logits into a probability distribution by exponentiating each one and normalizing by the total:

softmax(z)k=ezk∑j=1Kezj\text{softmax}(\mathbf{z})_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}
  • z\mathbf{z} — the vector of raw scores ("logits"), one per class, with no constraints on their range.
  • zkz_k — the logit for class kk, the one whose probability this expression computes.
  • KK — the number of classes.
  • ezke^{z_k} — exponentiating makes every term strictly positive, whatever sign zkz_k had.
  1. Set K=2 and this is exactly sigmoid

    With two classes, softmax reduces algebraically to σ(z1−z2)\sigma(z_1-z_2) — the same sigmoid from the binary chapter, just reframed as a difference of two scores instead of one score against a fixed 0.

  2. Shifting every logit by the same constant changes nothing

    softmax(z)=softmax(z+c)\text{softmax}(\mathbf{z}) = \text{softmax}(\mathbf{z}+c) for any constant cc — it cancels between every numerator and the shared denominator. In practice, implementations subtract max⁡(z)\max(\mathbf{z}) before exponentiating purely to avoid overflow, with no effect on the result.

  3. A ratio of two logits is not the ratio of two probabilities

    Doubling a logit does not double its probability — the exponential and the shared normalization mean the mapping from logits to probabilities is nonlinear throughout.

Play

Same three logits, one dial: temperature. Divide every logit by TT before the softmax. T<1T<1 sharpens the distribution toward one confident class; T>1T>1 flattens it toward uniform — the ranking of classes never changes, only how sure the model sounds.

Worked example

Logits z=[2,0,−1]z = [2, 0, -1] for cat, dog, bird:

  1. Exponentiate each logit
    • e2≈7.389e^2 \approx 7.389
    • e0=1e^0=1
    • e−1≈0.368e^{-1}\approx0.368
  2. Sum them for the normalizer

    7.389+1+0.368=8.7577.389 + 1 + 0.368 = 8.757.

  3. Divide each by the sum
    • p(cat)=7.389/8.757≈0.844p(\text{cat}) = 7.389/8.757 \approx 0.844
    • p(dog)=1/8.757≈0.114p(\text{dog}) = 1/8.757 \approx 0.114
    • p(bird)=0.368/8.757≈0.042p(\text{bird}) = 0.368/8.757 \approx 0.042

    And 0.844+0.114+0.042=10.844+0.114+0.042 = 1 exactly, as it must.

Checkpoint

Three logits: z = [1, 1, 2] for cat, dog, bird. Which value is the probability of the bird class (the one with the highest logit)?

Pick a value to try it
Summary
softmax(z)k=ezk∑jezj\text{softmax}(\mathbf{z})_k = \frac{e^{z_k}}{\sum_j e^{z_j}}

Softmax is sigmoid's generalization to KK classes: exponentiate every logit to make it positive, then normalize so the whole vector sums to exactly 1. It's the standard way a linear classifier's raw scores become calibrated, comparable probabilities across any number of classes.