Part III — Probability, Information Theory & Bayesian Inference · Chapter 1

Probability: outcomes & distributions

Hook

If you had to describe how likely each outcome of something uncertain is, what rule do those likelihoods have to follow?

Intuition
25%A25%B25%C25%D

Drag any bar. Watch what happens to the others — the whole picture always fills exactly one bar's worth of height, no more, no less.

Formalize

That's a probability distribution: a number for every outcome, following two rules —

P(xi)0,iP(xi)=1P(x_i) \geq 0, \qquad \sum_i P(x_i) = 1
  • P(xi)P(x_i) — the probability assigned to outcome xix_i.
  • xix_i — one specific outcome, indexed by ii, ranging over all possible outcomes.
  1. No negative shares

    No outcome gets a negative share.

  2. Shares add up to the whole

    The shares always add up to the whole thing.

Play
25%A25%B25%C25%D
E[X] = 2.50

Reshape the distribution and watch the expected value — the long-run average outcome — track the shift:

E[X]=iP(xi)xiE[X] = \sum_i P(x_i)\, x_i
Worked example

With P=(0.1,0.2,0.3,0.4)P = (0.1, 0.2, 0.3, 0.4) over outcomes worth (1,2,3,4)(1,2,3,4):

  1. Weight each outcome by its probability
    0.1(1)=0.1,0.2(2)=0.4,0.3(3)=0.9,0.4(4)=1.60.1(1) = 0.1, \quad 0.2(2) = 0.4, \quad 0.3(3) = 0.9, \quad 0.4(4) = 1.6
  2. Add them up for the expected value
    E[X]=0.1+0.4+0.9+1.6=3.0E[X] = 0.1+0.4+0.9+1.6 = 3.0
Checkpoint

Reshape the distribution until E[X] reads (approximately) 2.0.

25%A25%B25%C25%D
E[X] = 2.50
Drag a bar to try it
Summary
iP(xi)=1,E[X]=iP(xi)xi\sum_i P(x_i) = 1, \qquad E[X] = \sum_i P(x_i)\, x_i

A distribution is just a set of non-negative shares that add up to one. The expected value is what you'd average over many draws from it.