Dropout & stochastic depth

Click resample. Each time, a different random half of these neurons goes dark — and the ones that survive get scaled up to compensate. The output jumps around from draw to draw, but it's never systematically off from the true, undropped value.

Dropout randomly zeroes each neuron independently with probability $p$, then rescales the survivors by $\frac{1}{1-p}$ ("inverted dropout"):

• $\hat y$ — this layer's output after dropout is applied.
• $p$ — the dropout rate: the probability any given neuron is zeroed out.
• $v_i$ — the weight connecting surviving neuron $i$ to the output.
• $h_i$ — surviving neuron $i$'s activation.

1. The rescaling keeps the output unbiased

   This isn't arbitrary — it's exactly what keeps the expected value of $\hat y$, averaged over every possible random mask, equal to the true undropped output $\sum_i v_i h_i$, for any dropout rate $p$ at all.

Drag the rate up and resample. At higher rates, fewer neurons survive each draw and the output swings harder — but the expected active count $8(1-p)$ tells you exactly how many you should expect on average, even though any one draw might have more or fewer.

1. Average many random draws

   Over 200,000 independent draws at $p=0.5$, the mean output lands within about 1% of the true undropped value — confirming the unbiasedness the rescaling is designed to guarantee.

2. Look at a single draw instead

   Any single draw can differ from the true value by more than the true value's own size: dropout genuinely injects substantial per-step noise.

3. See why that noise is the point

   A network trained this way can't rely on any specific neuron or fixed combination of neurons always being present, so it's forced to spread what it learns across many redundant paths instead of memorizing through one fragile one.