Semantic segmentation & the U-Net

Encoding: each pool keeps only the block maximum, throwing the rest away.

A tiny image, pooled down twice to a single number, then upsampled back out with nothing else to go on. Step through the trace and watch exactly how much detail each pooling step throws away — and how little a plain upsample can do to bring it back.

A U-Net is an encoder-decoder: the encoder repeatedly pools an image down toward a small bottleneck, and the decoder upsamples it back out to full resolution. A skip connection carries each encoder stage's own feature map across to the matching decoder stage, so the decoder has real spatial detail to combine with, not just a coarse upsampled guess:

• $\text{decode}_i$ — the decoder's output at resolution level $i$ (matching the encoder's resolution at that same level).
• $\text{skip}_i$ — the encoder's own feature map at level $i$, saved before it was pooled down further.
• $\text{combine}$ — how the upsampled signal and the skip are merged (concatenation and a conv in a real U-Net; a plain average here, to keep the arithmetic exact by hand).

1. Pooling is lossy, on purpose

   Each $2\times2$ max pool keeps only a block's maximum — the other three values are gone the moment the encoder moves to the next stage. That's what makes the bottleneck cheap to compute over.

2. Upsampling alone can't undo that loss

   Nearest-neighbor upsampling only repeats each value into a bigger block. Chained twice from a $1\times1$ bottleneck, it can only ever produce one repeated number across the whole image.

3. The skip connection is the only path detail can take back

   Because $\text{skip}_i$ was saved before the corresponding pooling step, it still has the detail that step discarded — the decoder can only recover what the skip connection hands it directly.

The same $4\times4$ input, decoded two ways: one path only ever saw the bottleneck's single number, the other had the encoder's own feature maps to combine with at every stage. The reconstruction error tells the same story the heatmaps show.

Input:

1. Encoder: two poolings to a bottleneck

   Taking the max of each $2\times2$ block:

   • Top-left block $\begin{bmatrix}1&3\\4&2\end{bmatrix}$: max $=4$
   • Top-right block $\begin{bmatrix}6&2\\8&5\end{bmatrix}$: max $=8$
   • Bottom-left block $\begin{bmatrix}7&1\\0&5\end{bmatrix}$: max $=7$
   • Bottom-right block $\begin{bmatrix}3&9\\2&6\end{bmatrix}$: max $=9$

   giving $\text{skip}_1=\begin{bmatrix}4&8\\7&9\end{bmatrix}$. Pooling that once more gives a bottleneck of just $9$ — the single largest pixel in the whole image.

2. Without skip connections: a uniform blur

   Upsampling $9$ twice produces a $4\times4$ grid of nothing but $9$s. Squared error against the original input, averaged over all $16$ pixels: $\text{MSE}=31.75$.

3. With skip connections: real detail comes back

   First, average the upsampled bottleneck ($9$ everywhere) with $\text{skip}_1$'s own four values:

   $$\text{decode}_1 = \begin{bmatrix}(9+4)/2 & (9+8)/2\\(9+7)/2 & (9+9)/2\end{bmatrix} = \begin{bmatrix}6.5&8.5\\8&9\end{bmatrix}$$

   Upsampling that once more and averaging with the original input recovers real detail, in place of a flat $9$ either way:

   • the top-left pixel becomes $(1+6.5)/2=3.75$
   • the pixel at row 4, column 3 becomes $(2+9)/2=5.5$

   Averaged over all 16 pixels: $\text{MSE}=5.421875$.

4. The skip connection alone accounts for the entire gap

   Both decoders start from the exact same bottleneck. The only difference is whether $\text{skip}_0$ and $\text{skip}_1$ ever entered the computation — and that alone cuts reconstruction error by $31.75/5.421875\approx5.86\times$.