Weight initialization (Xavier & He/Kaiming)

This is a genuine 6-layer network — 8 inputs, 8 outputs per layer, fixed random weights — with every weight multiplied by one shared scale. Drag it small: the signal shrinks by roughly the same fraction every layer, and six layers of shrinking compounds into almost nothing left.

Each layer's output is a sum of $\text{fan-in}$ random-weighted terms. Averaged over random weights, that sum's typical size scales like $\sqrt{\text{fan-in}}\cdot\text{scale}$ times the input's size. For the signal to stay roughly the same size layer after layer — neither shrinking nor growing — that factor needs to sit near $1$:

• scale — the multiplier applied to every randomly-drawn initial weight.
• fan-in — the number of inputs feeding into a neuron or layer.

1. Xavier/Glorot: scale by the layer's own size

   Pick the initial weight scale from the layer's own size, not a fixed constant, so six layers or sixty behave the same way at the start.

The dashed lines mark a healthy band. Push the scale toward 1 and the curve rockets upward, doubling or more with every layer — the "exploding" failure mode. This chapter's dataset has fan-in 8, so $1/\sqrt{8}\approx0.354$ is the scale that keeps the curve inside the band the whole way across.

With fan-in $8$:

1. Too small — scale 0.05

   About seven times smaller than ideal ($0.05$ vs. $\approx0.354$). Each layer's signal scales by roughly $\sqrt{\text{fan-in}}\cdot\text{scale} = \sqrt8\times0.05\approx0.141$ — a shrink, not a hold. Compounding that six times: $0.141^6\approx0.000008$, taking the RMS (short for Root Mean Square) activation from about $1.5$ at the input to under $0.0001$ by layer 6 — a network that's already numerically dead before a single gradient step.

2. Too large — scale 1.0

   Roughly three times too large ($1.0$ vs. $\approx0.354$). Each layer's signal now scales by $\sqrt8\times1.0\approx2.83$ — a growth factor well above $1$. Compounding that six times: $2.83^6\approx514$, pushing the same six layers up past $400$.

3. Just right — the ideal scale ≈ 0.354

   Every one of the six layers stays within about $0.4\times$ to $2\times$ of the input's own size — no compounding in either direction.