LINEAR-ALGEBRA · Unit 14 · Video 3 · Interactive Practice

No Perfect Line? How Projection Finds the Best Fit

IKey Formulas

Formula	Name	Description
$A^T A \hat{x} = A^T b$	Normal equation	Converts unsolvable $Ax = b$ into a solvable square system
$p = A\hat{x}$	Projection	Closest point in the column space of $A$ to $b$
$e = b - A\hat{x}$	Error vector	Perpendicular to the column space: $A^T e = 0$
$\lVert e \rVert^2 = \sum (b_i - p_i)^2$	Squared error	The quantity minimized by least squares

IIVisualization 1 — Minimizing squared error

No line hits all three points; least squares picks the line whose total squared error is smallest.

💡 The line $b = t$ passes through two of the points exactly, yet its total squared error is $1$ ; the best fit accepts a small error at every point for a far lower total of $\tfrac{1}{6}$ .

IIIVisualization 2 — Projection makes the error perpendicular

The projection of $b$ onto the column space is the one point where the error runs perpendicular to it.

x 0.00

💡 The same test governs the video's 3D case: projecting $b = (1,2,2)$ onto the plane spanned by $(1,1,1)$ and $(1,2,3)$ leaves an error with $A^T e = 0$ .

IVVisualization 3 — How data reshapes the best fit

Drag the data points and the normal equation recomputes the fit — yet $A^TA$ never changes.

AᵀA = [[3, 6], [6, 14]] (fixed)

💡 The residuals always sum to zero because $A$ 's column of ones forces the error to have no component along it — the intercept absorbs the average.

VQuiz Questions

Question 1

For the data points $(1,1)$ , $(2,2)$ , $(3,2)$ with model $b = C + Dt$ , the design matrix is $A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}.$ What is $A^T A$ ?

✅ Correct! The top-left entry sums the squares of the ones column ( $1^2+1^2+1^2=3$ ), and the bottom-right sums the squares of the $t$ -column ( $1^2+2^2+3^2=14$ ).

❌ Not quite. Entry $(i,j)$ of $A^T A$ is the dot product of column $i$ and column $j$ of $A$ . Try computing each entry carefully.

Show solution

Solution:

$A^T A = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}$

Entry $(1,1)$ : $1(1) + 1(1) + 1(1) = 3$
Entry $(1,2)$ : $1(1) + 1(2) + 1(3) = 6$
Entry $(2,1)$ : $1(1) + 2(1) + 3(1) = 6$
Entry $(2,2)$ : $1(1) + 2(2) + 3(3) = 1 + 4 + 9 = 14$

$A^T A = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}$

Question 2

True or False: If the system $Ax = b$ is overdetermined (more equations than unknowns), the least-squares error vector $e = b - A\hat{x}$ is always the zero vector.

✅ Correct! An overdetermined system generally has no exact solution, so $e \neq 0$ . The error measures how far $b$ is from the column space of $A$ .

❌ Not quite. If $e = 0$ then $A\hat{x} = b$ exactly — but we said there's no exact solution! Think about what 'overdetermined' means geometrically.

Show solution

Solution: False.

The whole reason we need least squares is that $b$ typically does not lie in the column space of $A$ when the system is overdetermined. If $e$ were zero, that would mean $A\hat{x} = b$ exactly, contradicting the fact that the system has no exact solution.

In our example, $e = \left(-\frac{1}{6},\; \frac{1}{3},\; -\frac{1}{6}\right)^T \neq 0$ .

The error is zero only in the special case where the overdetermined system happens to be consistent (all equations are simultaneously satisfiable).

Question 3

For the best-fit line $b = \frac{2}{3} + \frac{1}{2}t$ through the points $(1,1)$ , $(2,2)$ , $(3,2)$ , what is the total squared error $\lVert e \rVert^2$ ?

Hint: Compute the prediction at each $t$ , find each residual, square them, and sum.

✅ Correct! The minimum total squared error is $1/6 \approx 0.167$ . No line can do better for these three points.

❌ Not quite. Start by computing each prediction: at $t=1$ , $\hat{b} = 2/3 + 1/2 = 7/6 \approx 1.167$ . Then find each residual (actual minus predicted), square, and sum.

Show solution

Solution:

Compute the prediction and residual at each data point:

$t$	Actual $b$	Predicted $\hat{b} = \frac{2}{3} + \frac{1}{2}t$	Residual $e$	$e^2$
1	1	$\frac{2}{3} + \frac{1}{2} = \frac{7}{6}$	$1 - \frac{7}{6} = -\frac{1}{6}$	$\frac{1}{36}$
2	2	$\frac{2}{3} + 1 = \frac{5}{3}$	$2 - \frac{5}{3} = +\frac{1}{3}$	$\frac{1}{9} = \frac{4}{36}$
3	2	$\frac{2}{3} + \frac{3}{2} = \frac{13}{6}$	$2 - \frac{13}{6} = -\frac{1}{6}$	$\frac{1}{36}$

$\lVert e \rVert^2 = \frac{1}{36} + \frac{4}{36} + \frac{1}{36} = \frac{6}{36} = \frac{1}{6}$

Question 4

To fit a parabola $b = C + Dt + Et^2$ to data at $t = 1, 2, 3$ , what would the design matrix $A$ be?

Each row corresponds to one data point, and each column to one unknown coefficient.

✅ Correct! Each new term in the model adds a column to $A$ : ones for the intercept, $t$ for linear, $t^2$ for quadratic. The recipe generalizes to any number of terms.

❌ Not quite. Each row is one data point. The columns hold the values of $1$ , $t$ , and $t^2$ evaluated at that point's $t$ -value.

Show solution

Solution:

For the model $b = C + Dt + Et^2$ , substitute each data point's $t$ -value:

$t = 1$ : $C(1) + D(1) + E(1^2) = C + D + E$ → row $[1,\; 1,\; 1]$
$t = 2$ : $C(1) + D(2) + E(2^2) = C + 2D + 4E$ → row $[1,\; 2,\; 4]$
$t = 3$ : $C(1) + D(3) + E(3^2) = C + 3D + 9E$ → row $[1,\; 3,\; 9]$

$A = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \end{bmatrix}$

Column 1 = ones (intercept), Column 2 = $t$ -values (linear), Column 3 = $t^2$ -values (quadratic).

With 3 equations and 3 unknowns, this system is square and (if $A$ is invertible) has an exact solution — the parabola passes through all three points!

Solved: 0 / 4