LINEAR-ALGEBRA · Unit 13 · Video 3 · Interactive Practice

No Exact Answer? Why Multiplying by Aᵀ Turns Unsolvable Systems into Solvable Ones

IKey Formulas

Formula	Name	When to Use
$A^T A \hat{x} = A^T b$	Normal Equations	Transform unsolvable $Ax = b$ into solvable system
$\hat{x} = (A^T A)^{-1} A^T b$	Least-Squares Solution	When $A$ has linearly independent columns
$A$ is $m \times n \Rightarrow A^T A$ is $n \times n$	Dimension Rule	Rectangular system → Square system
$A^T A$ invertible $\iff$ columns of $A$ independent	Invertibility Test	Determines if $\hat{x}$ is unique

IIVisualization 1 — Projection onto the column space

With $\text{Col}(A)$ drawn as a line, the shortest residual $b - p$ is the one that meets $\text{Col}(A)$ at a right angle.

Estimate x̂ 0.00

💡 When $b$ already lies in $\text{Col}(A)$ , this shortest residual is $0$ — then $Ax = b$ has an exact solution and $\hat{x}$ solves it outright.

IIIVisualization 2 — When $A^T A$ goes singular

As the columns of $A(t)$ slide into dependence at $t = 1$ , $\det(A^T A) = 26(1 - t)^2$ falls to $0$ and $A^T A$ loses its inverse.

Dependence t 0.00

💡 $A^T A$ is invertible exactly when the columns of $A$ are independent — so the uniqueness of $\hat{x}$ stands or falls with that independence.

IVVisualization 3 — Least squares minimizes the error

The normal-equations solution is the single line that makes the total squared residual $\sum r_i^2$ as small as possible.

Intercept c₀ 2.00 Slope c₁ 1.00

💡 GPS works this way: your phone fits its position to more satellite equations than the unknowns require, letting the normal equations average out each signal's noise.

VQuiz Questions

Question 1 · Dimensions of the Normal Equations

If $A$ is a $7 \times 3$ matrix (7 equations, 3 unknowns), what size is $A^T A$ ?

✅ Correct! $(3 \times 7)(7 \times 3) = 3 \times 3$ . The normal equations always give an $n \times n$ square system!

❌ Not quite. If $A$ is $m \times n$ , then $A^T$ is $n \times m$ . Multiply the inner dimensions: $(n \times m)(m \times n) = n \times n$ .

Show solution

Solution:

$A$ is $7 \times 3$ , so $A^T$ is $3 \times 7$ . Therefore:

$A^T A = \underbrace{(3 \times 7)}_{A^T} \cdot \underbrace{(7 \times 3)}_{A} = (3 \times 3)$

Rule: If $A$ is $m \times n$ , then $A^T A$ is always $n \times n$ — square, matching the number of unknowns, not the number of equations.

Question 2 · Symmetry of $A^T A$

True or False: The matrix $A^T A$ is always symmetric, regardless of the shape or entries of $A$ .

✅ Correct! $(A^T A)^T = A^T(A^T)^T = A^T A$ . Always symmetric, for any matrix $A$ !

❌ Not quite. Try computing $(A^T A)^T$ step by step using $(XY)^T = Y^T X^T$ .

Show solution

Solution: True.

Take the transpose of $A^T A$ :

$(A^T A)^T = A^T (A^T)^T = A^T A$

This uses two rules:

$(XY)^T = Y^T X^T$ (reverse order rule)
$(A^T)^T = A$ (double transpose cancels)

The transpose of $A^T A$ equals itself — that's the definition of symmetric! This holds for any matrix $A$ of any shape.

In the video's example: $A^T A = \begin{bmatrix}3 & 8\\8 & 30\end{bmatrix}$ — both off-diagonals are 8. ✓

Question 3 · Computing det(AᵀA)

Given $A = \begin{bmatrix}1 & 1\\1 & 2\\1 & 5\end{bmatrix}$ , what is $\det(A^T A)$ ?

(Hint: first compute $A^T A$ , then use $\det\!\begin{bmatrix}a & b\\c & d\end{bmatrix} = ad - bc$ .)

✅ Correct! $3 \times 30 - 8 \times 8 = 90 - 64 = 26 \neq 0$ . Invertible, unique $\hat{x}$ !

❌ Not quite. First find $A^T A = \begin{bmatrix}3 & 8\\8 & 30\end{bmatrix}$ , then compute $\det = 3(30) - 8(8)$ .

Show solution

Solution:

Step 1 — Compute $A^T A$ :

$A^T = \begin{bmatrix}1 & 1 & 1\\1 & 2 & 5\end{bmatrix}$

$A^T A = \begin{bmatrix}1\!\cdot\!1+1\!\cdot\!1+1\!\cdot\!1 & 1\!\cdot\!1+1\!\cdot\!2+1\!\cdot\!5\\1\!\cdot\!1+2\!\cdot\!1+5\!\cdot\!1 & 1\!\cdot\!1+2\!\cdot\!2+5\!\cdot\!5\end{bmatrix} = \begin{bmatrix}3 & 8\\8 & 30\end{bmatrix}$

Step 2 — Determinant:

$\det(A^T A) = 3 \times 30 - 8 \times 8 = 90 - 64 = 26$

Since $26 \neq 0$ , $A^T A$ is invertible → unique $\hat{x}$ exists!

Why the wrong answers are tempting:

90 = only $ad$ (forgot to subtract $bc$ )
64 = only $bc$ (forgot the $ad$ term)
0 = would mean dependent columns (they're independent here)

Question 4 · When Does Uniqueness Fail?

If $A = \begin{bmatrix}1 & 3\\1 & 3\\1 & 3\end{bmatrix}$ , what happens when you form the normal equations $A^T A \hat{x} = A^T b$ ?

✅ Correct! Dependent columns → $\det(A^T A) = 81 - 81 = 0$ → singular → no unique $\hat{x}$ .

❌ Not quite. Notice column 2 is $3 \times$ column 1 — the columns are linearly dependent. What does the key theorem say about dependent columns and $A^T A$ ?

Show solution

Solution:

Column 2 of $A$ is $\begin{bmatrix}3\\3\\3\end{bmatrix} = 3 \cdot \begin{bmatrix}1\\1\\1\end{bmatrix}$ = 3 × Column 1. The columns are linearly dependent!

$A^T A = \begin{bmatrix}1&1&1\\3&3&3\end{bmatrix}\begin{bmatrix}1&3\\1&3\\1&3\end{bmatrix} = \begin{bmatrix}3 & 9\\9 & 27\end{bmatrix}$

$\det = 3 \times 27 - 9 \times 9 = 81 - 81 = 0$

$A^T A$ is singular (not invertible). The normal equations still have solutions, but they are not unique.

This confirms the key theorem from the video: $A^T A$ is invertible if and only if the columns of $A$ are linearly independent.

Why the other options are wrong:

$A^T A$ can always be formed, regardless of $A$ 's shape — it's $n \times n$ .
$A^T A$ is $2 \times 2$ here (not $3 \times 3$ ), since $A$ has 2 columns.
Invertibility is not guaranteed — it depends on column independence.

Solved: 0 / 4

No Exact Answer? Why Multiplying by Aᵀ Turns Unsolvable Systems into Solvable Ones

IKey Formulas

IIVisualization 1 — Projection onto the column space

IIIVisualization 2 — When ATAA^T AATA goes singular

IVVisualization 3 — Least squares minimizes the error

VQuiz Questions

IIIVisualization 2 — When $A^T A$ goes singular