LINEAR-ALGEBRA · Unit 15 · Video 3 · Interactive Practice

One Solution, Two Pictures: Why Your Best-Fit Line Is a Shadow in Disguise

IKey Formulas

Formula	Name	Description
$A^T A \hat{x} = A^T b$	Normal equations	Solves the least-squares problem
$\mathbf{p} = A\hat{x}$	Projection	Closest vector in $\text{Col}(A)$ to $\mathbf{b}$
$\mathbf{e} = \mathbf{b} - \mathbf{p}$	Error vector	Must satisfy $\mathbf{e} \perp \text{Col}(A)$
$A^T \mathbf{e} = \mathbf{0}$	Orthogonality condition	Error perpendicular to every column of $A$

IIVisualization 1 — Best Fit in Data Space

No line hits all three points; the best fit is the one that minimizes the total squared residual $\sum e_i^2$ .

C (intercept) 0.50 D (slope) 1.00

IIIVisualization 2 — Projection onto the Column Space

$\mathbf{p}=C\mathbf{a}_1+D\mathbf{a}_2$ is $\mathbf{b}$ 's shadow on the plane $\text{Col}(A)$ ; the error $\mathbf{e}=\mathbf{b}-\mathbf{p}$ is shortest exactly when it meets the plane at a right angle.

C (weight on a₁) 0.50 D (weight on a₂) 0.50 rotate 130°

IVVisualization 3 — The Error Landscape

$\text{SSE}(C,D)$ is a convex bowl, and the normal equations jump straight to its single lowest point.

C (intercept) 0.50 D (slope) 1.00

VQuiz Questions

Question 1

For data points $(1,1)$ , $(2,2)$ , $(3,2)$ with model $y = C + Dt$ , we write the system as $A\mathbf{x} = \mathbf{b}$ . What is the right-hand side vector $\mathbf{b}$ ?

✅ Correct! The vector $\mathbf{b}$ collects the $y$ -values from each data point.

❌ Not quite. Each equation is $C + Dt_i = y_i$ , so $\mathbf{b}$ is the column of $y$ -values.

Show solution

Solution:

Substituting each data point $(t_i, y_i)$ into $y = C + Dt$ :

$C + D(1) = 1, \quad C + D(2) = 2, \quad C + D(3) = 2$

In matrix form:

$\underbrace{\begin{pmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{pmatrix}}_{A} \underbrace{\begin{pmatrix} C \\ D \end{pmatrix}}_{\mathbf{x}} = \underbrace{\begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}}_{\mathbf{b}}$

The right-hand side $\mathbf{b} = (1, 2, 2)$ collects the $y$ -values.

Common errors:

$(1, 2, 3)$ = the $t$ -values = second column of $A$
$(1, 1, 1)$ = the column of ones = first column of $A$
$(2/3, 1/2)$ = the solution $\hat{\mathbf{x}}$ , not $\mathbf{b}$

Question 2

True or False: If $\mathbf{p} \cdot \mathbf{e} = 0$ (the projection is perpendicular to the error), then we are guaranteed to have the least-squares solution.

✅ Correct! The condition $\mathbf{p} \cdot \mathbf{e} = 0$ is necessary but not sufficient. We need $A^T\mathbf{e} = \mathbf{0}$ .

❌ Not quite. Being perpendicular to $\mathbf{p}$ alone doesn't guarantee perpendicularity to the entire column space plane.

Show solution

Solution: False

Checking $\mathbf{p} \cdot \mathbf{e} = 0$ only verifies perpendicularity to one vector ( $\mathbf{p}$ ).

The least-squares condition requires:

$A^T\mathbf{e} = \mathbf{0}$

This means $\mathbf{e}$ must be perpendicular to every vector in the column space — both $\mathbf{a}_1$ and $\mathbf{a}_2$ , and all their linear combinations.

Many vectors in $\mathbb{R}^3$ can be perpendicular to $\mathbf{p}$ without being perpendicular to the entire plane. Only the vector perpendicular to the whole column space minimizes $\|\mathbf{e}\|$ .

Question 3

The optimal error vector is $\mathbf{e} = (-1/6, \; 1/3, \; -1/6)$ and the second column of $A$ is $\mathbf{a}_2 = (1, 2, 3)$ .

What is $\mathbf{e} \cdot \mathbf{a}_2$ ?

✅ Correct! The dot product is zero, confirming $\mathbf{e} \perp \mathbf{a}_2$ as required by $A^T\mathbf{e} = \mathbf{0}$ .

❌ Not quite. Compute carefully: $(-1/6)(1) + (1/3)(2) + (-1/6)(3)$ . Try converting all terms to sixths.

Show solution

Solution:

$\mathbf{e} \cdot \mathbf{a}_2 = \left(-\frac{1}{6}\right)(1) + \left(\frac{1}{3}\right)(2) + \left(-\frac{1}{6}\right)(3)$

Converting to sixths:

$= -\frac{1}{6} + \frac{4}{6} - \frac{3}{6} = \frac{-1 + 4 - 3}{6} = \frac{0}{6} = 0$

This confirms $\mathbf{e} \perp \mathbf{a}_2$ , which is one of the two conditions in $A^T\mathbf{e} = \mathbf{0}$ . (The other condition, $\mathbf{e} \cdot \mathbf{a}_1 = 0$ , also holds.)

Question 4

In the "two pictures" framework, the column space $\text{Col}(A)$ in the vector space picture corresponds to what in the data space picture?

✅ Correct! Each point in Col( $A$ ) corresponds to the fitted values of a line $y = C + Dt$ . The whole column space represents all possible linear fits.

❌ Not quite. Think about what $A\mathbf{x}$ produces as you vary $C$ and $D$ — each choice gives a different line and different fitted values.

Show solution

Solution:

The column space $\text{Col}(A)$ consists of all vectors $C\mathbf{a}_1 + D\mathbf{a}_2$ as $C$ and $D$ vary. Each such vector gives fitted values:

$(C + D, \; C + 2D, \; C + 3D)$

These are the values of the line $y = C + Dt$ evaluated at $t = 1, 2, 3$ .

So $\text{Col}(A)$ = the set of all possible fitted lines $y = C + Dt$ .

The full two-picture correspondence:

Data Space	Vector Space
Data points $(y_1, y_2, y_3)$	Vector $\mathbf{b}$
All possible lines $y = C + Dt$	Column space $\text{Col}(A)$
Fitted values on best-fit line	Projection $\mathbf{p}$
Vertical residuals	Error vector $\mathbf{e}$
Minimize $\sum e_i^2$	Minimize $\\|\mathbf{e}\\|^2$

Solved: 0 / 4