The perceptron & the XOR problem

Same two clusters as the SVM — short for Support Vector Machine — chapter. Take a step: one point gets checked against the current line. If it's already on the right side, nothing happens. If it's wrong, the line moves — just enough to nudge that one point onto its correct side.

This is the perceptron, one of the oldest learning algorithms. It's a single artificial neuron: a weight vector $w$ and bias $b$ predict $\hat y = \text{sign}(w \cdot x + b)$. Whenever a prediction is wrong, it corrects itself by exactly the term that caused the error:

• $w$ — the weight vector, the perceptron's current line direction.
• $b$ — the bias, shifting the line away from the origin.
• $\eta$ — the learning rate, controlling how big a correction each mistake causes.
• $y$ — the true label of the point just checked, $+1$ or $-1$.
• $x$ — the input point's coordinates.

1. No calculus needed

   Whenever the prediction is wrong (with $y$ the true label, $+1$ or $-1$), the rule above nudges $w$ and $b$ directly toward correctness — no calculus, no loss function to minimize, just "if wrong, shift toward being right."

Watch $w$ and $b$ directly. They only change on the two steps that actually hit a misclassified point — every other step leaves them untouched. The line isn't being optimized; it's being corrected, one mistake at a time.

Starting at $w=(0,0)$, $b=0$:

1. First point — a mistake

   $(3,6)$ is class A (label $-1$), but $\text{sign}(0) = +1$ — wrong. The update: $w \leftarrow (0,0) + 1\cdot(-1)\cdot(3,6) = (-3,-6)$, $b \leftarrow 0 + 1\cdot(-1) = -1$.

2. Next three points — no change

   Already correctly classified by this new line.

3. Point five — another mistake

   $(5,4)$ is class B ($+1$), but the current line predicts $-1$ — wrong again: $w \leftarrow (-3,-6) + (5,4) = (2,-2)$, $b \leftarrow -1+1=0$.

4. Converged

   From here, all eight points — checked again in a full second pass — are already correct. Just two corrections, out of sixteen checks, and it's converged to $w=(2,-2)$, $b=0$: the line $y=x$.