LINEAR-ALGEBRA · Unit 14 · Video 2 · Interactive Practice

Why Perpendicularity Alone Gives You the Projection Matrix

IKey Formulas

Formula Name Description
A⊤Ax^=A⊤bA^\top A \hat{x} = A^\top b Normal Equation Perpendicularity condition e⊥C(A)e \perp C(A) in matrix form
x^=(A⊤A)−1A⊤b\hat{x} = (A^\top A)^{-1} A^\top b Coefficients Combination weights for the projection
P=A(A⊤A)−1A⊤P = A(A^\top A)^{-1} A^\top Projection Matrix Sends any bb to its projection; satisfies P2=PP^2 = P and P⊤=PP^\top = P
P=a a⊤a⊤aP = \dfrac{a\, a^\top}{a^\top a} 1-D Special Case Projection onto the line through column vector aa

IIVisualization 1 — Projection onto a Line

The projection pp is the one point on the line where the residual b−pb-p meets it at a right angle.

IIIVisualization 2 — Projection onto a Plane

The columns a1=(1,0,1)a_1=(1,0,1) and a2=(0,1,1)a_2=(0,1,1) span a plane, and the residual e=b−pe=b-p is perpendicular to both at once.

💡 The residual lives in the left null space N(A⊤)N(A^\top) — the orthogonal complement of the column space C(A)C(A).

IVVisualization 3 — Idempotency: P2=PP^2 = P

Once bb lands on the subspace, projecting a second time moves nothing: P2b=PbP^2 b = Pb.

VVisualization 4 — The 1-D Projection Matrix

For a single direction aa, P=a a⊤a⊤aP=\dfrac{a\,a^\top}{a^\top a} stays symmetric and idempotent at every angle.

VIQuiz Questions

Question 1

Starting from the normal equation A⊤Ax^=A⊤bA^\top A \hat{x} = A^\top b, we solve for x^\hat{x} and then compute the projection vector pp.

Which formula correctly gives pp?

✅ Correct! The projection vector is p = Ax̂ = A(AᵀA)⁻¹Aᵀb. You multiply the coefficients x̂ by A to get the actual vector in the column space.

❌ Not quite. Remember: x̂ gives the coefficients, but the projection vector p = Ax̂ lives in the column space — you must multiply x̂ by A.

Show solution

Solution:

From the normal equation, solve for x^\hat{x}: x^=(A⊤A)−1A⊤b\hat{x} = (A^\top A)^{-1} A^\top b

The projection is p=Ax^p = A\hat{x}, so multiply on the left by AA: p=A(A⊤A)−1A⊤bp = A(A^\top A)^{-1} A^\top b

Why the other options are wrong:

  • (A⊤A)−1A⊤b(A^\top A)^{-1}A^\top b — this is x^\hat{x} (the coefficients), not the projected vector pp.
  • A⊤(AA⊤)−1AbA^\top(AA^\top)^{-1}Ab — the factors are in the wrong order; this is not derived from the normal equation.
  • A−1A⊤bA^{-1}A^\top b — this requires A−1A^{-1}, which does not exist when AA is rectangular.

Question 2

A student claims: "In P=A(A⊤A)−1A⊤P = A(A^\top A)^{-1}A^\top, I can split the inverse as (A⊤A)−1=A−1(A⊤)−1(A^\top A)^{-1} = A^{-1}(A^\top)^{-1}. Then A⋅A−1A \cdot A^{-1} cancels on the left and (A⊤)−1⋅A⊤(A^\top)^{-1} \cdot A^\top cancels on the right, giving P=IP = I."

True or False: This reasoning is valid when AA is a 3×23 \times 2 matrix with independent columns.

✅ Correct! A 3×2 matrix has no inverse — the cancellation is the most common mistake with the projection formula.

❌ Not quite. Can you invert a non-square matrix? A 3×2 matrix has no inverse, so the split is illegal.

Show solution

Solution: False.

The factorization (A⊤A)−1=A−1(A⊤)−1(A^\top A)^{-1} = A^{-1}(A^\top)^{-1} requires AA to be square and invertible.

A 3×23 \times 2 matrix has more rows than columns — it is not square. The inverse A−1A^{-1} simply does not exist, so the "cancellation" step is illegal.

P=IP = I is only correct when AA is square and invertible, meaning its columns span all of Rn\mathbb{R}^n. In that trivial case every vector bb is already in the column space, so projecting bb returns bb unchanged.

Question 3

Let a=[111]a = \begin{bmatrix}1\\1\\1\end{bmatrix}. Using the 1-D projection formula P=a a⊤a⊤aP = \dfrac{a\,a^\top}{a^\top a}, what is the (1,2)(1, 2) entry of PP?

✅ Correct! Since every component of a is 1 and aᵀa = 3, every entry of P = aaᵀ/3 equals 1/3.

❌ Not quite. Compute aaᵀ first (a 3×3 matrix of all 1s), then divide every entry by aᵀa = 3.

Show solution

Solution:

a⊤a=12+12+12=3a^\top a = 1^2 + 1^2 + 1^2 = 3

a a⊤=[111][111]=[111111111]a\,a^\top = \begin{bmatrix}1\\1\\1\end{bmatrix}\begin{bmatrix}1&1&1\end{bmatrix} = \begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}

P=13[111111111]P = \frac{1}{3}\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}

Every entry of PP equals 13\frac{1}{3}, so the (1,2)(1, 2) entry is 13\boxed{\tfrac{1}{3}}.

Question 4

If PP is a projection matrix onto a proper subspace (not the full space) and you compute P2bP^2 b for some vector bb, the result equals:

✅ Correct! This is idempotency: P² = P. Projecting a vector that's already on the subspace leaves it unchanged.

❌ Not quite. Think geometrically: if a vector is already sitting on the subspace, what does projecting it again do?

Show solution

Solution:

Compute P2P^2:

P2=[A(A⊤A)−1A⊤][A(A⊤A)−1A⊤]P^2 = \bigl[A(A^\top A)^{-1}A^\top\bigr]\bigl[A(A^\top A)^{-1}A^\top\bigr]

The interior product simplifies because (A⊤A)−1(A⊤A)=In(A^\top A)^{-1}(A^\top A) = I_n:

P2=A(A⊤A)−1(A⊤A)(A⊤A)−1⏟= InA⊤=A(A⊤A)−1A⊤=PP^2 = A(A^\top A)^{-1} \underbrace{(A^\top A)(A^\top A)^{-1}}_{= \, I_n} A^\top = A(A^\top A)^{-1}A^\top = P

Therefore P2b=PbP^2 b = Pb for every vector bb.

Geometric meaning: Once bb is projected onto the subspace, it's already there — projecting again changes nothing. This property is called idempotency.

Solved: 0 / 4