Part III — Probability, Information Theory & Bayesian Inference · Chapter 5

Gaussian, Bernoulli, Poisson & Dirichlet distributions

Hook

A coin flip, a count of rare events, a bell curve, a probability vector that's itself uncertain — four completely different-looking situations, and machine learning reaches for a named distribution to model every one of them.

Intuition

Toggle between the four. Each one has an exact formula behind it, and every bar (or curve point) on screen is that formula evaluated at a specific value — nothing here is approximated or simulated.

Formalize

Each distribution answers a different shape of question:

Bernoulli: P(X=1)=pPoisson: P(X=k)=e−λλkk!Gaussian: f(x)=1σ2πe−(x−μ)22σ2\text{Bernoulli: } P(X{=}1)=p \qquad \text{Poisson: } P(X{=}k)=\frac{e^{-\lambda}\lambda^k}{k!} \qquad \text{Gaussian: } f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}
  • pp — the Bernoulli's single parameter: the probability of outcome 1.
  • λ\lambda — the Poisson's rate parameter: the expected count.
  • μ,σ\mu, \sigma — the Gaussian's mean and standard deviation.

A Dirichlet distribution is one level up: instead of putting mass on numbers, it puts mass on probability vectors themselves. Its expected vector has a strikingly simple form:

E[pi]=αi∑jαj\mathbb{E}[p_i] = \frac{\alpha_i}{\sum_j \alpha_j}
  • αi\alpha_i — the concentration parameter for category ii (a pseudo-count of prior observations).
  • E[pi]\mathbb{E}[p_i] — the expected probability the Dirichlet assigns to category ii.
  1. Bernoulli is the atom

    Every other discrete distribution here is built by combining or generalizing Bernoulli trials.

  2. Poisson counts, Gaussian measures

    Poisson lives on non-negative integers (counts); the Gaussian lives on the whole real line (measurements) and is fully determined by just its first two moments.

  3. Dirichlet is a distribution over distributions

    Where Bernoulli/Poisson/Gaussian describe one random number, the Dirichlet describes an entire random categorical distribution — exactly the object a classifier's softmax output is trying to estimate.

Play

Poisson's bars and the Gaussian's curve both come from a single closed-form line of code — the same determinism that made the coin-flip Bernoulli exact also makes e−2λk/k!e^{-2}\lambda^k/k! and the bell-curve formula exact, just with more decimal places to track by hand.

Worked example
  1. Poisson(λ=2): compute P(X=0)
    P(X=0)=e−2⋅200!=e−2≈0.1353P(X{=}0) = \frac{e^{-2}\cdot 2^0}{0!} = e^{-2} \approx 0.1353
  2. Poisson(λ=2): compute P(X=1) and P(X=2) — and notice they tie
    P(X=1)=e−2⋅211!=2e−2≈0.2707P(X=2)=e−2⋅222!=4e−22=2e−2≈0.2707P(X{=}1) = \frac{e^{-2}\cdot 2^1}{1!} = 2e^{-2} \approx 0.2707 \qquad P(X{=}2) = \frac{e^{-2}\cdot 2^2}{2!} = \frac{4e^{-2}}{2} = 2e^{-2} \approx 0.2707

    When λ\lambda is a whole number, P(X=λ−1)P(X{=}\lambda{-}1) and P(X=λ)P(X{=}\lambda) are always exactly equal — the distribution has two tied modes.

  3. Dirichlet(α=[2,3,5]): compute the expected vector
    E[p]=[210,310,510]=[0.2, 0.3, 0.5]\mathbb{E}[p] = \left[\frac{2}{10}, \frac{3}{10}, \frac{5}{10}\right] = [0.2,\ 0.3,\ 0.5]

    The concentration parameters' own proportions become the expected probabilities — no extra machinery needed.

Checkpoint

For Poisson(λ=2), what is P(X=2)?

Pick a value to try it
Summary
P(X=1)=pP(X=k)=e−λλkk!f(x)=1σ2πe−(x−μ)22σ2E[pi]=αi∑jαjP(X{=}1)=p \qquad P(X{=}k)=\frac{e^{-\lambda}\lambda^k}{k!} \qquad f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} \qquad \mathbb{E}[p_i]=\frac{\alpha_i}{\sum_j\alpha_j}

Almost every probabilistic model in this course is one of these four shapes wearing a different hat: a classifier's output layer is Bernoulli or categorical, a count-based likelihood is Poisson, a continuous noise model is Gaussian, and a prior over class probabilities is Dirichlet.