Multivariable-Calculus · Unit 8 · Video 4 · Interactive Practice

Least Squares: Finding the Best Line Means Minimizing a Function of a and b

IKey Formulas

Formula	Name	Meaning
$y_i - (ax_i + b)$	Vertical deviation	measured $y$ minus the $y$ the line predicts at $x_i$
$D(a, b) = \sum_{i=1}^{n} \big(y_i - (ax_i + b)\big)^2$	Total square deviation	one number for each line; data $x_i, y_i$ fixed
$D(a, b) = (1 - b)^2 + (3 - a - b)^2 + (4 - 2a - b)^2$	Video example: $(0, 1), (1, 3), (2, 4)$	$D(1, 1) = 2$ , $\ D(2, 1) = 1$
$\frac{\partial D}{\partial a} = 0, \quad \frac{\partial D}{\partial b} = 0$	Critical point of $D$	where the best $a$ and $b$ must lie

Key Insight: The unknowns are the coefficients $a$ and $b$ , not $x$ and $y$ : once the measured $x_i, y_i$ are plugged in, $D$ is a function of $a$ and $b$ alone, and the best-fit line is the point $(a, b)$ where $D$ is smallest.

IIVisualization 1 — Every Line Is a Point $(a, b)$

Each choice of $(a, b)$ is one line through the fixed data, and $D(a, b)$ scores it with one number.

💡 Challenge: get $D$ below $1$ , the better of the video's two lines. Finding the exact minimum, by solving $\frac{\partial D}{\partial a} = \frac{\partial D}{\partial b} = 0$ , is the next video.

IIIVisualization 2 — Vertical Deviations Survive a Change of Units

Measuring $x$ in tenths of a second instead of seconds moves the perpendicular-distance best line but not the vertical one.

Unit of x 1 s

IVVisualization 3 — Largest, Average or Squared Deviation?

One bad measurement pulls hardest on the line that minimizes the largest deviation.

VQuiz Questions

Problem 1 · Total Square Deviation of One Line

Given: data points $(1, 4)$ , $(2, 2)$ , $(3, 3)$ and the line $y = x + 1$ (so $a = 1$ , $b = 1$ ) — find $D(1, 1) = \sum \big(y_i - (ax_i + b)\big)^2$ .

✅ Correct! The deviations are

2

-1

-1

, and their squares add to

4 + 1 + 1 = 6

❌ Not quite. The signed deviations

2 - 1 - 1

do add to

0

; that cancellation is exactly why each deviation is squared before adding.

❌ Not quite.

4 = 2 + 1 + 1

is the sum of the sizes of the deviations; least squares adds their squares.

❌ Not quite.

16 = (2 + 1 + 1)^2

squares the total; square each deviation first, then add.

❌ Not quite. Compute each deviation

y_i - (x_i + 1)

, square it, then add.

Show solution

The line predicts $x_i + 1$ , that is $2$ , $3$ , $4$ at $x = 1, 2, 3$ . The deviations (measured minus predicted) are

4 - 2 = 2, \qquad 2 - 3 = -1, \qquad 3 - 4 = -1.

D(1, 1) = 2^2 + (-1)^2 + (-1)^2 = 4 + 1 + 1 = 6.

The signed deviations add to $0$ , which is why the raw sum is useless: points above and below the line cancel.

Problem 2 · What Are the Unknowns?

Given: to fit $y = ax + b$ to measured data, a student minimizes $D(a, b) = \sum_{i=1}^{n} \big(y_i - (ax_i + b)\big)^2$ by setting $\frac{\partial D}{\partial x} = 0$ and $\frac{\partial D}{\partial y} = 0$ . Which statement is right?

✅ Correct!

D

is a function of

a

and

b

alone, so its critical point solves

\frac{\partial D}{\partial a} = \frac{\partial D}{\partial b} = 0

❌ Not quite.

x

and

y

are not solved for: their values

x_i, y_i

are given as data. The unknowns are the slope

a

and intercept

b

❌ Not quite. The

x_i

and

y_i

are measured numbers; we cannot move them. Only

a

and

b

are free.

❌ Not quite.

D(a, b) = 0

would need every point exactly on one line, which real data almost never allows; we minimize

D

instead.

Show solution

Once the measurements are plugged in, every $x_i$ and $y_i$ in $D(a, b) = \sum \big(y_i - (ax_i + b)\big)^2$ is a number. What remains free is the pair of coefficients describing the line, so $D$ is a function of two variables, $a$ and $b$ .

A minimum of a function of two variables is a critical point, where both partial derivatives vanish:

\frac{\partial D}{\partial a} = 0, \qquad \frac{\partial D}{\partial b} = 0.

Setting $D = 0$ asks for a line through every point, which generally does not exist; the least-squares line makes $D$ as small as possible, not zero.

Problem 3 · Expand the Video's D(a, b)

Given: the video's data $(0, 1)$ , $(1, 3)$ , $(2, 4)$ , for which $D(a, b) = (1 - b)^2 + (3 - a - b)^2 + (4 - 2a - b)^2$ — expand $D$ as a polynomial in $a$ and $b$ .

What is the coefficient of $a^2$ ?

What is the constant term?

✅ Correct!

D(a, b) = 5a^2 + 6ab + 3b^2 - 22a - 16b + 26

, and it reproduces

D(1, 1) = 2

and

D(2, 1) = 1

❌ Check the $a^2$ coefficient. The

a^2

terms come from

(-a)^2

and

(-2a)^2

; the first bracket has no

a

❌ Check the $a^2$ coefficient.

9 = (0 + 1 + 2)^2

squares the sum of the

x_i

; each bracket contributes

x_i^2

, so add the squares

0^2 + 1^2 + 2^2

❌ Check the $a^2$ coefficient.

3

is the number of points, which is the coefficient of

b^2

, not

a^2

❌ Check the constant term. Set

a = b = 0

: each bracket reduces to

y_i

, so the constant is a sum of squares of the

y_i

❌ Check the constant term.

64 = (1 + 3 + 4)^2

squares the total; the constant is

1^2 + 3^2 + 4^2

Show solution

Expand each bracket:

(1 - b)^2 = 1 - 2b + b^2

(3 - a - b)^2 = 9 + a^2 + b^2 - 6a - 6b + 2ab

(4 - 2a - b)^2 = 16 + 4a^2 + b^2 - 16a - 8b + 4ab

Add them:

D(a, b) = 5a^2 + 6ab + 3b^2 - 22a - 16b + 26.

In general the coefficient of $a^2$ is $\sum x_i^2 = 0 + 1 + 4 = 5$ and the constant is $\sum y_i^2 = 1 + 9 + 16 = 26$ .

Check: $D(1, 1) = 5 + 6 + 3 - 22 - 16 + 26 = 2$ and $D(2, 1) = 20 + 12 + 3 - 44 - 16 + 26 = 1$ , matching the video.

Problem 4 · A Line Through the Origin

Given: data $(1, 3)$ , $(2, 5)$ , $(3, 5)$ and a model $y = ax$ forced through the origin, so $D(a) = \sum \big(y_i - ax_i\big)^2$ has a single unknown — find the $a$ that minimizes $D$ , and the minimum value.

Which $a$ minimizes $D(a)$ ?

What is the minimum value of $D$ ?

✅ Correct!

D'(a) = 0

gives

a = \frac{\sum x_i y_i}{\sum x_i^2} = \frac{28}{14} = 2

, and the deviations

1

1

-1

give

D(2) = 3

❌ Check $a$ . Differentiate:

D'(a) = -2\sum x_i(y_i - ax_i)

, and solve

D'(a) = 0

❌ Check $a$ .

\tfrac{13}{6} = \tfrac{\sum y_i}{\sum x_i}

is a natural guess, but it is not where

D'(a) = 0

: it gives

D = \tfrac{61}{18} \approx 3.39

❌ Check $a$ .

\tfrac{5}{3}

is the slope to the last point alone; the least-squares

a

uses all three.

❌ Check the minimum. Evaluate

D

at the minimizing

a

: compute each deviation

y_i - ax_i

, square, add.

❌ Check the minimum.

D = 0

would need all three points on one line through the origin;

3/1 \ne 5/2

, so they are not.

Show solution

With one unknown, this is one-variable minimization:

D(a) = (3 - a)^2 + (5 - 2a)^2 + (5 - 3a)^2.

D'(a) = -2(3 - a) - 4(5 - 2a) - 6(5 - 3a) = -2\big(\textstyle\sum x_i y_i - a\sum x_i^2\big) = -2(28 - 14a).

Setting $D'(a) = 0$ gives $a = \frac{28}{14} = 2$ . Since $D''(a) = 28 > 0$ , this is a minimum.

At $a = 2$ the predictions are $2$ , $4$ , $6$ , so the deviations are $1$ , $1$ , $-1$ and

D(2) = 1^2 + 1^2 + (-1)^2 = 3.

The full line $y = ax + b$ has two unknowns, so the same idea needs two partial derivatives set to zero.

Solved: 0 / 4

Least Squares: Finding the Best Line Means Minimizing a Function of a and b

IKey Formulas

IIVisualization 1 — Every Line Is a Point (a,b)(a, b)(a,b)

IIIVisualization 2 — Vertical Deviations Survive a Change of Units

IVVisualization 3 — Largest, Average or Squared Deviation?

VQuiz Questions

IIVisualization 1 — Every Line Is a Point $(a, b)$