LINEAR-ALGEBRA · Unit 16 · Video 3 · Interactive Practice

Subtract Until Perpendicular: The Gram-Schmidt Process

IKey Formulas

Formula	Name	Description
$\text{proj}_{\mathbf{a}}(\mathbf{b}) = \dfrac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \cdot \mathbf{a}$	Vector projection	Shadow of $\mathbf{b}$ along $\mathbf{a}$
$\mathbf{B} = \mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b})$	Perpendicular residual	Part of $\mathbf{b}$ orthogonal to $\mathbf{a}$
$\mathbf{v}_k^{\perp} = \mathbf{v}_k - \displaystyle\sum_{j=1}^{k-1} \frac{(\mathbf{v}_j^{\perp})^T \mathbf{v}_k}{(\mathbf{v}_j^{\perp})^T \mathbf{v}_j^{\perp}} \cdot \mathbf{v}_j^{\perp}$	Gram-Schmidt (k-th step)	Subtract all previous shadows
$\mathbf{q}_k = \dfrac{\mathbf{v}_k^{\perp}}{\lVert \mathbf{v}_k^{\perp} \rVert}$	Normalization	Scale to unit length

IIVisualization 1 — Projection and Residual

Split $\mathbf{b}$ into a shadow along $\mathbf{a}$ and a leftover $\mathbf{B}$ — the leftover is perpendicular to $\mathbf{a}$ wherever $\mathbf{b}$ lands.

The cancellation is structural, not a coincidence of this $\mathbf{b}$ : $\mathbf{a}^T(\mathbf{b} - \text{proj}_{\mathbf{a}}\mathbf{b}) = \mathbf{a}^T\mathbf{b} - \mathbf{a}^T\mathbf{b} = 0$ .

IIIVisualization 2 — The Ratio That Lands on Perpendicular

Only one scaling $t$ of $\mathbf{a}$ leaves a residual $\mathbf{b} - t\,\mathbf{a}$ perpendicular to $\mathbf{a}$ — where does the dot product hit zero?

Ratio (t) 0.50

$\mathbf{a}\cdot(\mathbf{b} - t\,\mathbf{a}) = \mathbf{a}\cdot\mathbf{b} - t\,\lVert\mathbf{a}\rVert^2$ is linear in $t$ , so it vanishes at exactly one ratio: $t^\star = \dfrac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}}$ .

IVVisualization 3 — Gram-Schmidt, Step by Step

Subtract the shadow, then rescale: a skewed pair $\mathbf{a}, \mathbf{b}$ becomes an orthonormal pair $\mathbf{q}_1, \mathbf{q}_2$ spanning the same plane.

Step 1

Step 1 — The skewed pair

\mathbf{a} = (3, 1),\quad \mathbf{b} = (1, 3)

\mathbf{a}\cdot\mathbf{b} = 3 + 3 = 6 \ne 0

(angle

\approx 53\degree

)

Step 2 — Project b onto a

\dfrac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}} = \dfrac{6}{10} = \tfrac{3}{5}

\text{proj} = \tfrac{3}{5}\mathbf{a} = (1.8,\ 0.6)

Step 3 — Subtract to get orthogonal B

\mathbf{B} = \mathbf{b} - \text{proj} = (1,3) - (1.8, 0.6) = (-0.8,\ 2.4)

\mathbf{A}\cdot\mathbf{B} = 3(-0.8) + 1(2.4) = 0\ \checkmark

Step 4 — Normalize to orthonormal

\mathbf{q}_1 = \dfrac{\mathbf{A}}{\lVert\mathbf{A}\rVert} = \dfrac{(3,1)}{\sqrt{10}} \approx (0.95,\ 0.32)

\mathbf{q}_2 = \dfrac{\mathbf{B}}{\lVert\mathbf{B}\rVert} \approx (-0.32,\ 0.95)

\lVert\mathbf{q}_1\rVert = \lVert\mathbf{q}_2\rVert = 1,\quad \mathbf{q}_1\cdot\mathbf{q}_2 = 0

In 2D both bases span the whole plane; the real payoff appears in higher dimensions, where each new vector stays a combination of the originals, so the subspace is preserved exactly while its directions turn orthonormal.

VQuiz

Question 1

In the video's worked example, $\mathbf{a} = (1, 1, 1)$ and $\mathbf{b} = (1, 0, 2)$ .

What is the projection ratio $\dfrac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}}$ ?

✅ Correct! Both the dot product and the squared norm equal 3, so the ratio is 3/3 = 1.

❌ Not quite. Compute numerator (a·b) and denominator (a·a) separately, then divide.

Show solution

Solution:

Numerator: $\mathbf{a}^T \mathbf{b} = 1 \cdot 1 + 1 \cdot 0 + 1 \cdot 2 = 1 + 0 + 2 = 3$

Denominator: $\mathbf{a}^T \mathbf{a} = 1^2 + 1^2 + 1^2 = 1 + 1 + 1 = 3$

Ratio: $\dfrac{3}{3} = 1$

This means we subtract exactly one full copy of $\mathbf{a}$ from $\mathbf{b}$ to get the perpendicular residual.

Question 2

True or False: After applying the Gram-Schmidt process, the new orthogonal vectors span a different subspace than the original vectors.

✅ Correct! Gram-Schmidt only changes the basis — the subspace itself is preserved.

❌ Not quite. Think about how each new vector is built: it's always a linear combination of the originals.

Show solution

Solution: False.

Gram-Schmidt preserves the subspace. Each new orthogonal vector is constructed as a linear combination of the original vectors:

$\mathbf{B} = \mathbf{b} - \frac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \cdot \mathbf{a}$

Since $\mathbf{B}$ is a combination of $\mathbf{a}$ and $\mathbf{b}$ , it lies in the same plane they span. Every point reachable by combining $\mathbf{a}$ and $\mathbf{b}$ is also reachable by combining $\mathbf{A}$ and $\mathbf{B}$ (or $\mathbf{q}_1$ and $\mathbf{q}_2$ ).

As the video states: "The plane hasn't moved. Only the signposts within it changed."

Question 3

Continuing the video's example: $\mathbf{A} = (1, 1, 1)$ , $\mathbf{b} = (1, 0, 2)$ , and the projection ratio is $1$ .

What is the residual vector $\mathbf{B} = \mathbf{b} - 1 \cdot \mathbf{A}$ ?

✅ Correct! And you can verify: A · B = 1(0) + 1(−1) + 1(1) = 0. Perpendicular! ✓

❌ Not quite. Subtract component by component: (1, 0, 2) − (1, 1, 1). Watch the signs carefully.

Show solution

Solution:

$\mathbf{B} = \mathbf{b} - 1 \cdot \mathbf{A} = (1, 0, 2) - (1, 1, 1)$

Component by component:

$1 - 1 = 0$
$0 - 1 = -1$
$2 - 1 = 1$

$\mathbf{B} = (0, -1, 1)$

Verification: $\mathbf{A} \cdot \mathbf{B} = 1(0) + 1(-1) + 1(1) = 0 - 1 + 1 = 0 \checkmark$

The dot product is zero, confirming $\mathbf{A}$ and $\mathbf{B}$ are orthogonal.

Question 4

In the Gram-Schmidt process applied to a set of 5 linearly independent vectors, how many projection terms must be subtracted when orthogonalizing the 5th vector?

✅ Correct! The k-th vector requires (k − 1) projection subtractions — one for each direction already established.

❌ Not quite. The k-th vector subtracts projections onto vectors 1 through (k − 1). How many terms is that when k = 5?

Show solution

Solution:

The general Gram-Schmidt formula for the $k$ -th vector is:

$\mathbf{v}_k^{\perp} = \mathbf{v}_k - \sum_{j=1}^{k-1} \frac{(\mathbf{v}_j^{\perp})^T \mathbf{v}_k}{(\mathbf{v}_j^{\perp})^T \mathbf{v}_j^{\perp}} \cdot \mathbf{v}_j^{\perp}$

For $k = 5$ , the sum runs from $j = 1$ to $j = 4$ , giving 4 projection subtractions.

The pattern:

Vector	Projections subtracted
$k = 1$	0 (accepted as is)
$k = 2$	1
$k = 3$	2
$k = 4$	3
$k = 5$	4

Each new vector must remove its shadow from every previously established orthogonal direction.

Solved: 0 / 4