Linear regression & ordinary least squares

Drag either end of the line. The dashed segments are each point's residual — how far it misses the line, straight up or down. A good fit is one where those misses are all small at once.

Squaring each residual before adding them up gives the standard measure of fit — the sum of squared errors (SSE):

• SSE — the sum of squared errors: how far the line's predictions miss the actual data, added up.
• $y_i$ — the actual (observed) value for data point $i$.
• $w$ — the line's slope.
• $x_i$ — the input value for data point $i$.
• $b$ — the line's intercept.

1. Every term becomes positive

   Squaring means misses in opposite directions can't cancel out — a point the line overshoots and a point it undershoots both add to the total instead of offsetting each other.

2. One large miss costs far more than several small ones

   Squaring also makes a big residual disproportionately expensive, so a line has to stay close to every point, not just close on average.

Drag toward a better angle and watch $w$, $b$, and the SSE update together — the line's slope and intercept are just two numbers, and SSE is one number summarizing how well those two numbers fit everything at once.

With $w=2, b=1$:

1. Compute the predictions

   $\hat y_i = 2x_i+1$ at each $x$:

   • $x=1$: $2(1)+1=3$
   • $x=3$: $2(3)+1=7$
   • $x=5$: $2(5)+1=11$
   • $x=7$: $2(7)+1=15$
   • $x=9$: $2(9)+1=19$

   Against actual values $3.5, 6.8, 10.5, 15.2, 18.5$.

2. Compute the residuals

   $y_i-\hat y_i$ at each point:

   • $3.5-3=0.5$
   • $6.8-7=-0.2$
   • $10.5-11=-0.5$
   • $15.2-15=0.2$
   • $18.5-19=-0.5$
3. Square and sum them

   $$\text{SSE} = 0.25+0.04+0.25+0.04+0.25 = 0.83$$

   Close, clean numbers — and already a good fit, without needing the exact optimum.