LINEAR-ALGEBRA · Interactive Practice | Unit 31 · Video 2

Left and Right Inverses (and the Projections Hiding in Between)

IKey Formulas

Formula	Name	When it works
$(A^T A)^{-1} A^T \, A = I$	Left-inverse	Tall matrix, full column rank ( $m > n$ , $r = n$ )
$A \, A^T (A A^T)^{-1} = I$	Right-inverse	Wide matrix, full row rank ( $n > m$ , $r = m$ )
$P = A (A^T A)^{-1} A^T$	Projection matrix	Same pieces, wrong order; $P^2 = P$ , $P^T = P$
$A^T A \, \hat{x} = A^T b$	Normal equations	Least squares: $A\hat{x} = Pb$

IIVisualization 1 — Which side gets the inverse?

A rectangular matrix inverts from only one side — the Gram matrix that stays full-rank picks which.

Matrix shape

IIIVisualization 2 — Projection onto the column space

$P$ sends $b$ to the nearest point of the column space — straight down, perpendicular.

b_x 2 b_y 3

💡 Because $P$ fixes every vector already in the column space, projecting twice changes nothing: $P^2 = P$ .

IVVisualization 3 — Same pieces, different order

The same three matrices give the identity in one order and a projection in the other.

VQuiz Questions

Question 1

Let $A$ be a tall matrix that is $m \times n$ with $m > n$ and full column rank ( $r = n$ ). Which one-sided inverse is guaranteed to exist?

✅ Correct! $A^T A$ is invertible, giving the left-inverse.

❌ Not quite. For a tall matrix it is $A^T A$ (not $A A^T$ ) that is invertible.

Show solution

Solution:

For a tall matrix with full column rank, the columns are independent, so the Gram matrix $A^T A$ (size $n \times n$ ) is invertible. This gives the left-inverse:

$(A^T A)^{-1} A^T \cdot A = (A^T A)^{-1}(A^T A) = I_n.$

A right-inverse would require $A A^T$ ( $m \times m$ ) to be invertible, but its rank is only $n < m$ , so it is singular. Hence only the left-inverse exists.

Question 2

A student claims: "Since the tall matrix $A$ has a left-inverse, it must also have a right-inverse — just multiply $A A^T (A A^T)^{-1}$ ."

True or False: This reasoning is correct.

✅ Correct! $A A^T$ has rank $n < m$ , so it is singular — no right-inverse.

❌ Not quite. $A A^T$ is $m \times m$ but rank $n < m$ , so it cannot be inverted.

Show solution

Solution:

The claim is False. For the tall case, $A A^T$ is $m \times m$ but its rank is only $n < m$ , so $A A^T$ is singular — its inverse $(A A^T)^{-1}$ does not exist.

The deeper reason: a rectangular matrix can never be inverted from both sides, because either $A$ or $A^T$ always has a nontrivial null space. Something always gets collapsed, and a collapse cannot be undone from both directions.

Question 3

For the projection matrix $P = A(A^T A)^{-1} A^T$ , which pair of properties always holds?

✅ Correct! Projections are symmetric and idempotent ( $P^2 = P$ ).

❌ Not quite. $P$ is symmetric, but projecting twice gives $P^2 = P$ , not $I$ .

Show solution

Solution:

A projection matrix satisfies two defining properties:

Symmetric: $P^T = P$ .
Idempotent: $P^2 = P$ (projecting twice is the same as projecting once — once $b$ is on the plane, it stays put).

It is not the identity ( $P \neq I$ ) and it is not invertible (it collapses the component perpendicular to the column space). For the example matrix, $P = \begin{bmatrix} 2/3 & -1/3 & 1/3 \\ -1/3 & 2/3 & 1/3 \\ 1/3 & 1/3 & 2/3 \end{bmatrix}$ indeed satisfies $P^T = P$ and $P^2 = P$ .

Question 4

An over-determined system $Ax = b$ has no exact solution. We solve the normal equations $A^T A \,\hat{x} = A^T b$ to get the least-squares solution $\hat{x}$ . What does $A\hat{x}$ equal?

✅ Correct! Least squares delivers $A\hat{x} = Pb$ , the projection of $b$ .

❌ Not quite. Since $Ax=b$ is unsolvable, $A\hat{x}$ is the projection $Pb$ , not $b$ itself.

Show solution

Solution:

Solving the normal equations gives $\hat{x} = (A^T A)^{-1} A^T b$ . Therefore

$A\hat{x} = A(A^T A)^{-1} A^T \, b = P b,$

which is exactly the projection of $b$ onto the column space of $A$ . Since $b$ is unreachable (no exact solution), $Pb$ is the closest reachable point — the best approximation. The Gram matrix $A^T A$ reappears here as the coefficient matrix of the normal equations, and it is invertible precisely because $A$ has full column rank.

Solved: 0 / 4