LINEAR-ALGEBRA · Interactive Practice | Unit 27 · Video 1

Three Guarantees: Why Positive Definiteness Survives Inversion, Addition, and A^TA

IKey Formulas

Concept	Statement	Tool to use
Positive definite (quadratic form)	$x^T A x > 0$ for all $x \neq 0$	Quadratic form
Inverse eigenvalue	$A x = \lambda x \implies A^{-1} x = \tfrac{1}{\lambda} x$	Eigenvalues
Sum closure	$x^T(A+B)x = x^T A x + x^T B x$	Quadratic form
$A^T A$ identity	$x^T A^T A x = \\|Ax\\|^2 \geq 0$	Norm identity
Strict positive definiteness for $A^T A$	Requires $A$ to have independent columns	Full column rank

IIVisualization 1 — The Reciprocal Map $\lambda \mapsto 1/\lambda$

For a positive-definite $A$ , every eigenvalue $\lambda > 0$ becomes $1/\lambda > 0$ for $A^{-1}$ — positivity survives inversion.

λ₁ of A 2.00 λ₂ of A 5.00

💡 As an eigenvalue approaches $0$ , its reciprocal grows without bound — yet never turns negative, so $A^{-1}$ stays positive definite.

IIIVisualization 2 — Sum of Two Positive-Definite Bowls

Adding two positive-definite forms adds their bowls; the sum is a steeper bowl whose level set stays a bounded ellipse.

A curvature (a) 2 B curvature (b) 3

IVVisualization 3 — When Is $A^{T}A$ Positive Definite?

$A^{T}A$ is positive definite exactly when the columns of $A$ are independent — otherwise it is only semi-definite.

col₂ #1 2 col₂ #2 1 col₂ #3 0

💡 With $\text{col}_1 = (1,2,3)$ fixed, the parallelogram's area equals $\sqrt{\det(A^{T}A)}$ — it collapses to $0$ precisely when the columns become dependent.

VQuiz Questions

Question 1

A symmetric matrix $A$ has eigenvalues $\{3, 4\}$ . What are the eigenvalues of $A^{-1}$ , and is $A^{-1}$ positive definite?

✅ Correct! Reciprocals of positive numbers are positive, so $A^{-1}$ stays positive definite.

❌ Not quite. Use $Ax = \lambda x \Rightarrow A^{-1}x = (1/\lambda)x$ . Eigenvalues invert, but signs stay the same.

Show solution

Solution:

If $Ax = \lambda x$ , then multiplying both sides by $A^{-1}$ gives $x = \lambda A^{-1} x$ , so

$A^{-1} x = \frac{1}{\lambda} x.$

So the eigenvalues of $A^{-1}$ are the reciprocals of the eigenvalues of $A$ : $\{1/3, 1/4\}$ .

Both reciprocals are positive (positive over positive). Since $A^{-1}$ is also symmetric (because $(A^{-1})^T = (A^T)^{-1} = A^{-1}$ ) and all its eigenvalues are positive, $A^{-1}$ is positive definite.

Question 2

True or False: If $A$ and $B$ are symmetric positive definite matrices, then the eigenvalues of $A + B$ are simply the sums of the eigenvalues of $A$ and $B$ .

✅ Correct! Eigenvalues don't add unless eigenvectors are shared. That's why we switch tools and use the quadratic form for the sum.

❌ Not quite. Eigenvalues of $A+B$ are NOT the sums of eigenvalues of $A$ and $B$ in general. This is precisely why we use the quadratic form for the sum closure proof.

Show solution

Solution:

False. Eigenvalues only add when $A$ and $B$ share the same eigenvectors. In general the eigenvalues of $A+B$ are something else entirely.

This is exactly why the eigenvalue tool fails for the sum closure property. We have to switch to the quadratic form:

$x^T(A+B)x = x^T A x + x^T B x > 0 + 0 = 0 \quad \text{for } x \neq 0.$

So $A+B$ is positive definite, even though we cannot say anything clean about its individual eigenvalues.

Question 3

Consider $A = \begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{bmatrix}$ . Which of the following best describes $A^T A$ ?

✅ Correct! Column 2 = 2·(Column 1), so the columns are dependent. $A^T A$ is symmetric and PSD, but $\det(A^T A) = 0$ .

❌ Not quite. Check whether the columns of $A$ are independent. If column 2 is a multiple of column 1, then $A^T A$ is singular.

Show solution

Solution:

The second column of $A$ is exactly twice the first column, so the columns are dependent (rank 1).

$A^T A$ is always symmetric: $(A^T A)^T = A^T (A^T)^T = A^T A$ .

We compute: $A^T A = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{bmatrix} = \begin{bmatrix} 14 & 28 \\ 28 & 56 \end{bmatrix}.$

The determinant is $14 \cdot 56 - 28^2 = 784 - 784 = 0$ , so $A^T A$ is singular.

The identity $x^T A^T A x = \|Ax\|^2 \geq 0$ guarantees positive semi-definiteness, but with dependent columns there exists a nonzero $x$ (for instance $x = (2, -1)^T$ ) such that $Ax = 0$ , killing strict positivity.

Question 4

The normal equations for least squares are $A^T A \hat{x} = A^T b$ . Why is having independent columns of $A$ the crucial requirement for a unique least-squares solution?

✅ Correct! Independent columns → $\|Ax\|^2 > 0$ for $x \neq 0$ → $A^T A$ positive definite → invertible → unique $\hat{x}$ .

❌ Not quite. $A$ is usually rectangular, so we can't invert $A$ directly. The trick is that $A^T A$ is square — and it's positive definite (hence invertible) exactly when the columns of $A$ are independent.

Show solution

Solution:

The chain of reasoning is:

Independent columns of $A$ $\Longleftrightarrow$ null space of $A$ is $\{0\}$ $\Longleftrightarrow$ $Ax = 0$ only when $x = 0$ .
The identity $x^T A^T A x = \|Ax\|^2$ is then strictly positive for every $x \neq 0$ .
Therefore $A^T A$ is positive definite — symmetric with all positive eigenvalues.
Positive definite matrices are invertible, so the normal equations have the unique solution

$\hat{x} = (A^T A)^{-1} A^T b.$

Note: $A$ itself is typically rectangular (more rows than columns), so we cannot invert $A$ directly. The whole point of $A^T A$ is to convert a tall rectangular matrix into a square positive definite one — provided the columns are independent.

Solved: 0 / 4

Three Guarantees: Why Positive Definiteness Survives Inversion, Addition, and ATA

IKey Formulas

IIVisualization 1 — The Reciprocal Map λ↦1/λ\lambda \mapsto 1/\lambdaλ↦1/λ

IIIVisualization 2 — Sum of Two Positive-Definite Bowls

IVVisualization 3 — When Is ATAA^{T}AATA Positive Definite?

VQuiz Questions

Three Guarantees: Why Positive Definiteness Survives Inversion, Addition, and A^TA

IIVisualization 1 — The Reciprocal Map $\lambda \mapsto 1/\lambda$

IVVisualization 3 — When Is $A^{T}A$ Positive Definite?