LINEAR-ALGEBRA · Unit 15 · Video 2 · Interactive Practice

No Perfect Answer? Why Squaring Your Errors Finds the Best-Fit Line

IKey Formulas

Formula	Name	Description
$y = C + Dt$	Best-fit line	Intercept $C$ , slope $D$
$E = \\|Ax - b\\|^2 = \sum e_i^2$	Total squared error	Sum of squared residuals
$A^T A \hat{x} = A^T b$	Normal equations	Solve for best-fit parameters $\hat{x}$
$e = b - A\hat{x} \perp \text{Col}(A)$	Orthogonality	Error is perpendicular to column space

IIVisualization 1 — The Least-Squares Minimum

The best fit for $(1,1)$ , $(2,2)$ , $(3,2)$ is the line whose squared residuals — drawn as literal squares — have the least total area.

💡 The line where that total area bottoms out is exactly the solution of the normal equations $A^T A\,\hat{x} = A^T b$ .

IIIVisualization 2 — Why Square the Errors?

Three ways to total the residuals as the slope $D$ varies — but only squaring gives a smooth curve with a single, calculus-findable minimum.

Slope D (C = 2/3 fixed) 0.50

💡 Setting the squared curve's slope to zero, $\frac{dE}{dD} = 0$ , is a linear equation — exactly what $A^T A\,\hat{x} = A^T b$ encodes.

IVVisualization 3 — Outlier Sensitivity

One rogue point can tilt the whole fit, because squaring magnifies its single large error.

Outlier y at t = 4 2.50

💡 Robust fits (least absolute deviations, Huber loss) blunt this by not squaring large errors — so always plot your data before trusting a formula.

VQuiz Questions

Question 1

For data points $(1, 1)$ , $(2, 2)$ , $(3, 2)$ , the system $Ax = b$ uses:

$A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}, \quad b = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}$

What is $A^T A$ ?

✅ Correct! Top-left = sum of ones = 3, off-diagonal = sum of t-values = 6, bottom-right = sum of t² = 1 + 4 + 9 = 14.

❌ Not quite. Each entry of A^T A is a dot product of columns of A. Try computing the bottom-right entry: 1² + 2² + 3² = ?

Show solution

Solution:

$A^T = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix}$

Multiply row by column:

Top-left: $1\!\cdot\!1 + 1\!\cdot\!1 + 1\!\cdot\!1 = 3$
Top-right: $1\!\cdot\!1 + 1\!\cdot\!2 + 1\!\cdot\!3 = 6$
Bottom-left: $1\!\cdot\!1 + 2\!\cdot\!1 + 3\!\cdot\!1 = 6$ (symmetric)
Bottom-right: $1\!\cdot\!1 + 2\!\cdot\!2 + 3\!\cdot\!3 = 1 + 4 + 9 = 14$

$A^T A = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}$

Question 2

The normal equations for the same data reduce to:

$3C + 6D = 5 \qquad \text{and} \qquad 6C + 14D = 11$

Solving by elimination, what is the least-squares best-fit line $y = C + Dt$ ?

✅ Correct! The best-fit line is $y = \frac{2}{3} + \frac{1}{2}t$ , with intercept $C = \frac{2}{3}$ and slope $D = \frac{1}{2}$ .

❌ Not quite. Try elimination: multiply the first equation by 2, then subtract from the second to isolate D.

Show solution

Solution:

From the normal equations: $3C + 6D = 5 \quad (1) \qquad 6C + 14D = 11 \quad (2)$

Step 1: Multiply $(1)$ by 2: $\;6C + 12D = 10$

Step 2: Subtract from $(2)$ : $(6C + 14D) - (6C + 12D) = 11 - 10$ , so $2D = 1$ , giving $D = \tfrac{1}{2}$ .

Step 3: Back-substitute into $(1)$ : $3C + 3 = 5$ , so $C = \tfrac{2}{3}$ .

$\boxed{y = \tfrac{2}{3} + \tfrac{1}{2}\,t}$

Question 3

True or False: The calculus approach (setting $\frac{\partial E}{\partial C} = 0$ and $\frac{\partial E}{\partial D} = 0$ ) yields different equations than the linear algebra normal equations $A^T A \hat{x} = A^T b$ .

✅ Correct! Both methods produce the same normal equations. Two roads, one destination — because both minimize the same quadratic objective.

❌ Not quite. Both methods are minimizing the same thing — the total squared error. Think about whether the same objective function can lead to different optimality conditions.

Show solution

Answer: False — both approaches give exactly the same equations.

Calculus route: Expand $E = (C + D - 1)^2 + (C + 2D - 2)^2 + (C + 3D - 2)^2$ , then set $\frac{\partial E}{\partial C} = 0$ and $\frac{\partial E}{\partial D} = 0$ . After simplification:

$3C + 6D = 5 \qquad 6C + 14D = 11$

Linear algebra route: Compute $A^T A \hat{x} = A^T b$ directly:

$\begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 5 \\ 11 \end{bmatrix}$

These are identical. The deep reason: differentiating a squared linear expression always produces a linear equation — exactly what $A^T A \hat{x} = A^T b$ encodes.

Question 4

A least-squares fit includes three well-behaved points with errors of about 1 unit each, plus one outlier with an error of 8 units. Approximately what fraction of the total squared error comes from the outlier alone?

✅ Correct! Since 8² = 64 dwarfs 3 × 1² = 3, the outlier alone accounts for roughly 96% of the total squared error.

❌ Not quite. Remember to square each error before comparing. The outlier contributes 8² = 64, while the three good points together contribute only 3.

Show solution

Solution: About 96%.

Compute squared errors:

Three good points: $1^2 + 1^2 + 1^2 = 3$
Outlier: $8^2 = 64$
Total: $3 + 64 = 67$

Outlier's share: $\dfrac{64}{67} \approx 95.5\% \approx 96\%$

Squaring amplifies large errors disproportionately. The outlier contributes about 96% of the total despite being only one of four data points. This is why one extreme value can dominate the entire least-squares fit.

Solved: 0 / 4