KL divergence & mutual information

P is fixed. Drag Q's bars around and watch the readout shrink toward zero as Q starts to look like P — and grow the more they disagree.

The KL divergence — short for Kullback–Leibler divergence — measures the extra bits it costs to describe data from $P$ using a code built for $Q$ instead of $P$'s own optimal code:

• $D_{KL}(P\Vert Q)$ — the divergence itself, in bits: how many extra bits $Q$'s code wastes on data actually drawn from $P$.
• $P(x)$ — the true probability of outcome $x$.
• $Q(x)$ — the model's (or approximate) probability of that same outcome $x$.
• $x$ — one outcome, ranging over everything either distribution assigns probability to.

1. Never negative, zero only at a perfect match

   $D_{KL}(P\Vert Q) \geq 0$ always, and it hits exactly $0$ only when $P=Q$ everywhere — the only way to waste zero bits on $Q$'s code is for $Q$ to already be the right code.

2. Not symmetric — direction matters

   $D_{KL}(P\Vert Q) \neq D_{KL}(Q\Vert P)$ in general. Despite measuring how "far apart" two distributions are, swapping which one is the reference changes the answer, so it isn't a true distance.

3. Related to cross-entropy

   Cross-entropy $H(P,Q) = -\sum_x P(x)\log_2 Q(x)$ always decomposes as $H(P,Q) = H(P) + D_{KL}(P\Vert Q)$ — Part II's cross-entropy loss chapter is exactly this $H(P,Q)$, with $P$ the true one-hot label.

Now watch both directions at once. $D_{KL}(P\Vert Q)$ and $D_{KL}(Q\Vert P)$ move together, roughly, but they're never quite equal — drag until they're close and notice they still don't match exactly.

With $P=(0.5,0.3,0.2)$ and $Q=(0.4,0.4,0.2)$:

1. KL divergence, one direction

   Compute each outcome's term separately:

   • $0.5\log_2\frac{0.5}{0.4}=0.5\log_2(1.25)\approx0.5(0.3219)\approx0.1610$
   • $0.3\log_2\frac{0.3}{0.4}=0.3\log_2(0.75)\approx0.3(-0.4150)\approx-0.1245$
   • The third term is exactly $0$ since $P$ and $Q$ agree there ($\log_2(1)=0$)

   Summing:

   $$D_{KL}(P\Vert Q) \approx 0.1610 - 0.1245 + 0 \approx 0.0365 \text{ bits}$$
2. The reverse direction gives a different number

   Same process, swapped:

   • $0.4\log_2\frac{0.4}{0.5}=0.4\log_2(0.8)\approx0.4(-0.3219)\approx-0.1288$
   • $0.4\log_2\frac{0.4}{0.3}=0.4\log_2(1.333)\approx0.4(0.4150)\approx0.1660$
   • The third term is again $0$

   Summing:

   $$D_{KL}(Q\Vert P) \approx -0.1288+0.1660+0 \approx 0.0372 \text{ bits}$$

   — close to the forward value here, but not identical, confirming the asymmetry.

3. Mutual information: KL applied to independence itself

   For a joint distribution over two binary variables, $P(X{=}0,Y{=}0){=}P(X{=}1,Y{=}1){=}0.4$ and $P(X{=}0,Y{=}1){=}P(X{=}1,Y{=}0){=}0.1$ — $X$ and $Y$ agree far more than chance. Both marginals are uniform, $(0.5,0.5)$, so $H(X)=H(Y)=1$ bit.

   The joint entropy sums the same $-p\log_2 p$ recipe over all four outcomes:

   • $-0.4\log_2(0.4)\approx0.5288$, for each of the two $0.4$ outcomes
   • $-0.1\log_2(0.1)\approx0.3322$, for each of the two $0.1$ outcomes

   $$H(X,Y) \approx 2(0.5288) + 2(0.3322) \approx 1.7219 \text{ bits}$$

   Mutual information is exactly the gap: $I(X;Y)=H(X)+H(Y)-H(X,Y)\approx 1+1-1.7219\approx 0.2781$ bits — equivalently, it's the KL divergence between the real joint distribution and the independent one its own marginals would predict. Zero mutual information means knowing $X$ tells you nothing about $Y$; here, it tells you a fair amount.