Information & entropy

Drag the bars from equal toward one outcome dominating. Once an outcome becomes almost certain, there's really nothing left to learn from watching the next draw — you already knew what was coming.

That "amount left to learn" is entropy, measured in bits —

• $H(X)$ — the entropy of the distribution, in bits.
• $P(x_i)$ — the probability of outcome $x_i$.
• $x_i$ — one specific outcome, indexed by $i$.

1. Zero when certain

   It's exactly zero when one outcome is certain.

2. Maximal when uniform

   It's largest exactly when every outcome is equally likely — uncertainty is maximized when you have the least information to favor any one outcome.

Drag toward uniform and watch entropy climb to its ceiling; drag toward certainty and watch it collapse toward zero.

With $P = (0.5, 0.25, 0.125, 0.125)$:

1. Compute each term, -p log₂ p

   • $0.5(1) = 0.5$
   • $0.25(2) = 0.5$
   • $0.125(3) = 0.375$
   • $0.125(3) = 0.375$

   Here the probabilities are all powers of two, so every term comes out to a whole number of bits.

2. Add them up for the entropy

   $$H(X) = 0.5+0.5+0.375+0.375 = 1.75 \text{ bits}$$