One of your data points is way off from the rest. Should it get to boss around your line as much as everyone else?
Notice the one point sitting far above the others. Drag the line near it, then away from it, and watch its dashed residual stretch and shrink far more dramatically than any of the rest.
Two ways to average those residuals into one number — Mean Squared Error (MSE) and Mean Absolute Error (MAE):
- MSE — Mean Squared Error: the average of the squared residuals.
- MAE — Mean Absolute Error: the average of the absolute residuals.
- — the number of data points.
- — the actual (observed) value for data point .
- — the model's predicted value for data point .
- MSE lets one big miss dominate
Squaring means a residual twice as large costs four times as much, so a single large miss can dominate the entire sum.
- MAE grows only proportionally
Absolute value scales linearly with the size of the miss, so a single outlier can't hijack the total the way it can under squared error.
Fit the five close-together points well, then watch both numbers. MAE settles down close to what it was without the outlier at all — MSE stays stubbornly large no matter where you put the line.
With , the outlier at has residual .
- Its leverage under squared error
Squared, that's — versus the other five residuals' squares summing to just , so the outlier accounts for over 99% of the total sum of squared errors (SSE).
- Its leverage under absolute error
In absolute terms it's against the other five's combined — still large, but only about 89% of the total.
- Compare the two
The same point, two very different amounts of leverage.
Fit the line so MAE drops below 4.0 — notice MSE won’t cooperate nearly as easily, no matter where you put the line.
MSE punishes large misses so heavily that a single outlier can dominate the fit; MAE treats every miss proportionally, making it far more robust to one bad point. The choice of loss function is a choice about which mistakes you consider expensive.