Multivariable-Calculus · Unit 8 · Video 5 · Interactive Practice

Least Squares Is Just Solving Linear Equations: Lines, Exponentials and Parabolas

IKey Formulas

Formula	Name	What you need
$D(a, b) = \sum_{i=1}^{n} (y_i - ax_i - b)^2$	Total square deviation	The data and a candidate line $y = ax + b$
$\begin{aligned} \left(\sum x_i^2\right) a + \left(\sum x_i\right) b &= \sum x_i y_i \\ \left(\sum x_i\right) a + n\,b &= \sum y_i \end{aligned}$	Best-fit line: $2 \times 2$ linear system	$\frac{\partial D}{\partial a} = 0$ and $\frac{\partial D}{\partial b} = 0$ , divided by $-2$
$y = c\,e^{ax} \iff \ln y = ax + b$ , with $c = e^{b}$	Exponential fit	The same system with $\ln y_i$ in place of $y_i$
$\left(\sum x_i^4\right) a + \left(\sum x_i^3\right) b + \left(\sum x_i^2\right) c = \sum x_i^2 y_i$	Best parabola $y = ax^2 + bx + c$ : first row of a $3 \times 3$ system	Each row down lowers every power of $x$ by one, ending in $n\,c = \sum x_i^0\, c$

Key Insight: the unknown coefficients enter the model linearly, so setting every partial derivative of $D$ to zero gives a linear system — the data only supply its numbers.

IIVisualization 1 — Data, Then Sums, Then Solve

Every coefficient of the $2 \times 2$ system is a sum computed from the data points.

💡 Challenge: place the three points so that $D = 0$ , then so that the best-fit slope is $a = 0$ although no two points share a height.

IIIVisualization 2 — Moore's Law on Two Scales

A straight line fitted to $\ln y$ is an exponential curve $y = c\,e^{at}$ on the ordinary scale.

Least squares on

IVVisualization 3 — The Best Parabola Is a 3 × 3 System

Adding the point $(3, 2)$ to the video's data bends the trend; a parabola needs sums up to $\sum x_i^4$ .

Step 1

Step 1 — Sums from the data

n = 4,\quad \sum x_i = 6,\quad \sum x_i^2 = 14

\sum x_i^3 = 36,\quad \sum x_i^4 = 98

\sum y_i = 10,\quad \sum x_i y_i = 17,\quad \sum x_i^2 y_i = 37

VQuiz Questions

Problem 1 · Build the 2 × 2 System

Given: the data points $(0, 2)$ , $(1, 2)$ , $(2, 5)$ — find the linear system whose solution is the best-fit line $y = ax + b$ .

✅ Correct! Subtracting gives

2a = 3

, so

a = \tfrac{3}{2}

b = \tfrac{3}{2}

: the best-fit line is

y = \tfrac{3}{2}x + \tfrac{3}{2}

❌ Count the b's. In

\sum (x_i a + b - y_i) = 0

one

b

appears for each data point, so the coefficient of

b

n = 3

, not

1

❌ Sum of squares, not square of the sum. The coefficient of

a

\sum x_i^2 = 0 + 1 + 4 = 5

, while

\left(\sum x_i\right)^2 = 9

❌ The right-hand sides are swapped. The row with

\sum x_i^2

comes from

\frac{\partial D}{\partial a}

, whose right side is

\sum x_i y_i = 12

; the row with

n\,b

has right side

\sum y_i = 9

❌ Not quite. Compute

\sum x_i

\sum x_i^2

\sum y_i

\sum x_i y_i

and place them in the normal equations.

Show solution

Here $n = 3$ and the four sums are

\sum x_i = 0 + 1 + 2 = 3,\qquad \sum x_i^2 = 0 + 1 + 4 = 5

\sum y_i = 2 + 2 + 5 = 9,\qquad \sum x_i y_i = 0 + 2 + 10 = 12

Placing them in $\left(\sum x_i^2\right) a + \left(\sum x_i\right) b = \sum x_i y_i$ and $\left(\sum x_i\right) a + n\,b = \sum y_i$ :

5a + 3b = 12,\qquad 3a + 3b = 9

Subtracting the second from the first gives $2a = 3$ , so $a = \tfrac{3}{2}$ ; then $\tfrac{9}{2} + 3b = 9$ gives $b = \tfrac{3}{2}$ . The best-fit line is $y = \tfrac{3}{2}x + \tfrac{3}{2}$ .

Problem 2 · Fitting an Exponential

Given: the data points $(0, e)$ , $(1, e^3)$ , $(2, e^4)$ and the model $y = c\,e^{ax}$ , fitted by least squares on $\ln y$ — find $a$ and $c$ .

What is a?

What is c?

✅ Correct! The fit is

y = e^{7/6}\,e^{3x/2}

: the best line

\ln y = \tfrac{3}{2}x + \tfrac{7}{6}

through the points

(x_i, \ln y_i)

, exponentiated.

❌ Only the intercept gets exponentiated. In

\ln y = ax + b

the slope of the line is

a

itself; it is

b = \ln c

that needs

c = e^{b}

❌ That is the intercept. Solving

5a + 3b = 11

3a + 3b = 8

gives slope

a = \tfrac{3}{2}

and intercept

b = \tfrac{7}{6}

❌ That line only uses two points.

\ln y

rises from

1

3

between the first two points, but the best line must account for all three:

a = \tfrac{3}{2}

❌ Check $a$ . Take logs first: the points become

(0, 1)

(1, 3)

(2, 4)

❌ That is $b = \ln c$ , not $c$ . The line fit returns

b = \tfrac{7}{6}

; undo the logarithm:

c = e^{7/6}

❌ Wrong direction.

b = \ln c

, so

c = e^{b}

, not

\ln b

❌ The fit need not pass through $(0, e)$ . At

x = 0

the best line gives

\ln y = \tfrac{7}{6}

, not the data value

\ln e = 1

❌ Check $c$ . Find the intercept

b

of the best line through

(x_i, \ln y_i)

, then

c = e^{b}

Show solution

Taking logs, $\ln y = \ln c + ax$ . With $b = \ln c$ this is the line $\ln y = ax + b$ , fitted to the points $(x_i, \ln y_i) = (0, 1)$ , $(1, 3)$ , $(2, 4)$ — the video's three points.

The sums are $n = 3$ , $\sum x_i = 3$ , $\sum x_i^2 = 5$ , $\sum \ln y_i = 8$ , $\sum x_i \ln y_i = 11$ , so

5a + 3b = 11,\qquad 3a + 3b = 8

Subtracting gives $2a = 3$ , so $a = \tfrac{3}{2}$ , and $\tfrac{9}{2} + 3b = 8$ gives $b = \tfrac{7}{6}$ . Then

c = e^{b} = e^{7/6} \approx 3.21,\qquad y = 3.21\,e^{3x/2}\ \text{(approximately)}

This fit minimizes the squared deviations of $\ln y$ , not of $y$ itself.

Problem 3 · Four Points, Full Fit

Given: the data points $(0, 1)$ , $(1, 1)$ , $(2, 2)$ , $(3, 4)$ — find the best-fit line and its total square deviation $D$ .

What is the best-fit line?

What is D at the best fit?

✅ Correct!

y = x + \tfrac{1}{2}

misses each point by exactly

\tfrac{1}{2}

, so

D = 4 \cdot \tfrac{1}{4} = 1

❌ Sum of squares, not square of the sum. The coefficient of

a

in the first row is

\sum x_i^2 = 14

; that line came from using

\left(\sum x_i\right)^2 = 36

❌ Count the b's. With four data points the second row is

6a + 4b = 8

, not

6a + b = 8

❌ The intercept is an unknown too.

a = \sum x_i y_i / \sum x_i^2

is the best line through the origin; the model

y = ax + b

needs both equations.

❌ Check the line. The system is

14a + 6b = 17

6a + 4b = 8

❌ Add up every point.

\tfrac{1}{4}

is one squared deviation; there are four of them.

❌ Square before adding. The deviations

\tfrac12, -\tfrac12, -\tfrac12, \tfrac12

do sum to

0

— that is the second normal equation — but

D

adds their squares.

❌ Square, don't take absolute values.

\sum |r_i| = 2

, but

D = \sum r_i^2

and

\left(\tfrac{1}{2}\right)^2 = \tfrac{1}{4}

❌ Check $D$ . Compute each deviation

y_i - (ax_i + b)

, square it, and add.

Show solution

Sums ( $n = 4$ ):

\sum x_i = 6,\quad \sum x_i^2 = 14,\quad \sum y_i = 8,\quad \sum x_i y_i = 0 + 1 + 4 + 12 = 17

System:

14a + 6b = 17,\qquad 6a + 4b = 8

Multiply the first by $2$ and the second by $3$ : $28a + 12b = 34$ and $18a + 12b = 24$ . Subtracting gives $10a = 10$ , so $a = 1$ , and $6 + 4b = 8$ gives $b = \tfrac{1}{2}$ .

Deviations $y_i - \left(x_i + \tfrac12\right)$ : $\tfrac{1}{2}$ , $-\tfrac{1}{2}$ , $-\tfrac{1}{2}$ , $\tfrac{1}{2}$ , so

D = \tfrac{1}{4} + \tfrac{1}{4} + \tfrac{1}{4} + \tfrac{1}{4} = 1

Problem 4 · A One-Coefficient Model

Given: the model $y = ax^2$ (no other terms) and the data points $(-1, 2)$ , $(1, 4)$ , $(2, 6)$ — find the least-squares value of $a$ .

✅ Correct!

D'(a) = 0

gives

\left(\sum x_i^4\right) a = \sum x_i^2 y_i

, so

a = \tfrac{30}{18} = \tfrac{5}{3}

: a

1 \times 1

linear system.

❌ That is $\sum y_i / \sum x_i^2$ . Differentiating

(y_i - ax_i^2)^2

brings out a factor

-x_i^2

, so each term is weighted by

x_i^2

a = \sum x_i^2 y_i / \sum x_i^4

❌ Averaging $y_i / x_i^2$ is not least squares. That treats every point as equally reliable for

a

; least squares weights each point by

x_i^2

❌ Sum of fourth powers, not square of the sum. The denominator is

\sum x_i^4 = 1 + 1 + 16 = 18

, not

\left(\sum x_i^2\right)^2 = 36

❌ Not quite. Write

D(a) = \sum (y_i - ax_i^2)^2

and set

D'(a) = 0

Show solution

The total square deviation is a function of the single unknown $a$ :

D(a) = \sum_{i=1}^{3} \left(y_i - a x_i^2\right)^2

By the chain rule, $D'(a) = \sum 2\left(y_i - a x_i^2\right)\left(-x_i^2\right)$ . Setting it to zero and dropping the $-2$ :

\left(\sum x_i^4\right) a = \sum x_i^2 y_i

With the data, $\sum x_i^4 = 1 + 1 + 16 = 18$ and $\sum x_i^2 y_i = 2 + 4 + 24 = 30$ , so

a = \frac{30}{18} = \frac{5}{3}

This is the first row of the parabola system with the $b$ and $c$ columns removed, because those coefficients are no longer in the model. Since $D''(a) = 2\sum x_i^4 > 0$ , it is a minimum.

Solved: 0 / 4