Train a CNN and inspect its learned feature filters

Four tiny 6×6 images, two classes: a vertical edge (the boundary runs top-to-bottom) and a horizontal edge (the boundary runs left-to-right). Two 3×3 filters, initialized to small random numbers, feed a small dense layer through ReLU (short for Rectified Linear Unit) and max pooling — the exact shape from Chapter 3, but every weight starts unlearned. Click train and watch the loss fall as the filters reorganize themselves into something that actually tells the two classes apart.

The architecture is unchanged from Chapter 3 — image → convolution → ReLU → pool → flatten → dense → softmax — but now there are two filters instead of one, and a real loss to minimize:

• $\mathcal{L}$ — the loss being minimized during training.
• $p_{\text{true class}}$ — the softmax probability the network assigns to the correct class.

1. This is cross-entropy loss

   It's the cross-entropy loss from Part II, computed over the softmax of the two output logits.

2. Backprop threads through every stage

   Backpropagation runs through every stage, in reverse order:

   1. The softmax gradient
   2. Back through the dense weights
   3. Back through max-pooling, which routes each gradient only to the position that actually won each window
   4. Back through ReLU, zero wherever the pre-activation wasn't positive
   5. Finally back into the two 3×3 kernels themselves

   The same chain rule as always, just longer.

Reset, then train one step at a time. Early on, the two random filters produce feature maps that don't separate the classes — predictions hover near 50%. After a few steps, watch one filter's weights drift toward something with a clear positive-to-negative split across a row or column — a shape it invented on its own, not copied from anywhere, because it's the shape that actually helps tell a vertical edge from a horizontal one.

1. Chapter 1's hand-designed kernel

   The vertical-edge kernel $\begin{pmatrix} -1 & 0 & 1 \\ -1 & 0 & 1 \\ -1 & 0 & 1 \end{pmatrix}$: every row sums to zero, and the sign flips from one side of the kernel to the other.

2. What a trained filter converges to

   A filter that learns to solve this task tends toward the same kind of structure — not the identical numbers, but the same shape of solution: something whose row-sums or column-sums cancel out for a constant patch, and don't cancel when a patch straddles an edge. Nobody told the network that; gradient descent found it because it's what reduces the loss.