Lesson 23: The Method of Least Squares II

Objectives

Given a dataset, represent least squares (the “target problem”) as a vector equation and as a matrix equation.
Evaluate a least-squares model at a given input and interpret the answer.
Use residual vectors and/or RMSEs to compare multiple linear models for the same dataset and determine the best fit.

1. Least Squares as Vector and Matrix Equations

Objective. Given a dataset, represent least squares (the “target problem”) as a vector equation and as a matrix equation.

Definition 1 (Target problem).

For $n$ data points $(x_1, y_1), \dots, (x_n, y_n)$ , the components of the input vector $\overline{\vphantom{b}x}$ are the inputs, and the components of the target vector $\overline{\vphantom{b}y}$ are the outputs. The intercept vector $\overline{\vphantom{b}1}_{n}$ has $n$ components, each equal to 1 (Lesson 19 Notes, section 3). The target problem is the problem of finding the scalars $m$ and $b$ that make the linear combination $m\overline{\vphantom{b}x} + b\overline{\vphantom{b}1}_{n}$ equal to the target vector $\overline{\vphantom{b}y}$ . As a vector equation and as a matrix equation, the target problem is

m\overline{\vphantom{b}x} + b\overline{\vphantom{b}1}_{n} = \overline{\vphantom{b}y}, \qquad\qquad \begin{pmatrix*}[r] \overline{\vphantom{b}x} & \overline{\vphantom{b}1}_{n} \end{pmatrix*}\begin{pmatrix*}[r] m \\ b \end{pmatrix*} = \overline{\vphantom{b}y}.

When the data points do not all lie on one line, neither equation has an exact solution. The least squares $m$ and $b$ make the residual vector $\overline{\vphantom{b}r} = \overline{\vphantom{b}y} - (m\overline{\vphantom{b}x} + b\overline{\vphantom{b}1}_{n})$ (Lesson 22 Notes, section 2) as short as possible.

Equation form. The intercept vector is required, because $b$ is a scalar and a scalar cannot be added to a vector.

Example 1

Write the target problem for the data below as a vector equation and as a matrix equation.

$x$	1	2	3	4	5
$y$	5	7	11	14	16

ShowHide solution

The target problem is $m\overline{\vphantom{b}x} + b\overline{\vphantom{b}1}_{n} = \overline{\vphantom{b}y}$ , here with $n = 5$ :

m\begin{pmatrix*}[r] 1 \\ 2 \\ 3 \\ 4 \\ 5 \end{pmatrix*} + b\begin{pmatrix*}[r] 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{pmatrix*} = \begin{pmatrix*}[r] 5 \\ 7 \\ 11 \\ 14 \\ 16 \end{pmatrix*}, \qquad\qquad \begin{pmatrix*}[r] 1 & 1 \\ 2 & 1 \\ 3 & 1 \\ 4 & 1 \\ 5 & 1 \end{pmatrix*}\begin{pmatrix*}[r] m \\ b \end{pmatrix*} = \begin{pmatrix*}[r] 5 \\ 7 \\ 11 \\ 14 \\ 16 \end{pmatrix*}.

Example 2

The MMAC dataset MonthlyUnemployment records the United States unemployment rate Rate, in percent, for 60 months, with Months numbering the months and month 1 for January 2010. Write the target problem as a vector equation and as a matrix equation, then find the least squares model of Rate as a function of Months.

ShowHide solution

With $n = 60$ data points and the target vector $\overline{\vphantom{b}\text{Rate}}$ , the target problem is

m\,\overline{\vphantom{b}\text{Months}} + b\overline{\vphantom{b}1}_{60} = \overline{\vphantom{b}\text{Rate}}, \qquad\qquad \begin{pmatrix*}[r] 1 & 1 \\ 2 & 1 \\ \vdots & \vdots \\ 60 & 1 \end{pmatrix*}\begin{pmatrix*}[r] m \\ b \end{pmatrix*} = \begin{pmatrix*}[r] 9.8 \\ 9.8 \\ \vdots \\ 5.6 \end{pmatrix*}.

In R, project() returns the least squares $b$ , then $m$ :

project(Rate ~ Months + 1, data = MonthlyUnemployment)
(Intercept)      Months
10.17802260 -0.07053626

The least squares model is

\text{Rate} = -0.07053626\,\text{Months} + 10.17802260.

Next: 2. Evaluating a Least Squares Model Lesson map

2. Evaluating a Least Squares Model

Objective. Evaluate a least-squares model at a given input and interpret the answer.

To evaluate the model $y = mx + b$ at an input, substitute the input for $x$ . To interpret the answer, state the input and the predicted output with units, for example “At month 36, the model predicts an unemployment rate of about 7.639%.”

Prediction, data value, and residual. At an input in the dataset, the model’s prediction is $mx + b$ and the data value is $y$ . The residual is the data value minus the prediction, $y - (mx + b)$ .

Evaluating in R. Evaluate the least squares model with the stored fit from the Lesson 22 Notes, section 2. A model built by hand, such as a line through two points, is stored the same way, with makeFun() and its own $m$ and $b$ .

Columns and entries in R. To pull one column of a dataset out as a vector, write dataset$column. To get entry $k$ of a vector v, write v[k].

Rounding. For the MMAC dataset Mortgage15YrAnnual, project() prints $b = 437.5255336$ and $m = -0.2155336$ . With the stored fit, the model predicts a 15-year mortgage rate of 4.734% in 2008:

cf = project(Rate ~ Year + 1, data = Mortgage15YrAnnual)
b = cf[1]
m = cf[2]
f = makeFun(m*Year + b ~ Year)
f(2008)
[1] 4.734071

The same model with its coefficients retyped as $-0.216$ and 437.526 predicts 3.798%, because the slope’s rounding error of 0.0004664 is multiplied by 2008.

Example 3

The least squares model of Rate as a function of Months for the MMAC dataset MonthlyUnemployment is

\text{Rate} = -0.07053626\,\text{Months} + 10.17802260.

Evaluate the model at month 36 and at month 65, and interpret each answer. Then compare the prediction at month 36 with the actual rate that month.

ShowHide solution

Store the least squares fit from project() at full precision, evaluate it at months 36 and 65, and subtract the month 36 prediction from the actual rate:

cf = project(Rate ~ Months + 1, data = MonthlyUnemployment)
b = cf[1]
m = cf[2]
f = makeFun(m*Months + b ~ Months)
f(36)
[1] 7.638717
f(65)
[1] 5.593166
Rate = MonthlyUnemployment$Rate
Rate[36]
[1] 7.9
Rate[36] - f(36)
[1] 0.2612828

At month 36, the model predicts an unemployment rate of about 7.639%. At month 65, the model predicts an unemployment rate of about 5.593%. The prediction at month 65 lies beyond the data, which end at month 60, and is less reliable than a prediction inside the data. The actual rate at month 36 was 7.9%, about 0.261 percentage points above the model’s prediction.

Next: 3. Comparing Linear Models Lesson map

3. Comparing Linear Models

Objective. Use residual vectors and/or RMSEs to compare multiple linear models for the same dataset and determine the best fit.

Criterion 2 (Better fit).

For linear models of the same dataset, the model with the shorter residual vector fits better. Equivalently, the model with the smaller root mean squared error (RMSE) fits better, because every model of the dataset has the same number of data points $n$ .

Compare every model by the same measure. When some models are given by residual length and others by RMSE, convert each residual length to an RMSE by dividing it by $\sqrt{n}$ (Lesson 22 Notes, section 3).

Different datasets. Rank models by RMSE only when they describe the same dataset, because each RMSE measures distance to its own data, in the units of its own output.

Example 4

Find the least squares model for the data below in R. Then compare it with the line through the first and last data points, by residual vector length and by RMSE.

$x$	1	2	3	4	5
$y$	5	7	11	14	16

ShowHide solution

Fit the least squares model, find the line through the first and last data points, then compare the two residual vectors by length and by RMSE. Store the data as vectors and run the method of least squares with project():

x = c(1, 2, 3, 4, 5)
y = c(5, 7, 11, 14, 16)
project(y ~ x + 1)
(Intercept)           x
1.9         2.9

The least squares model is $y = 2.9x + 1.9$ . Substitute $(1, 5)$ and $(5, 16)$ into the slope formula, then solve $5 = m(1) + b$ for the intercept:

m = \frac{16 - 5}{5 - 1} = 2.75, \qquad b = 5 - 2.75(1) = 2.25.

The residual vector of the least squares model is $\overline{\vphantom{b}r}_1$ , and the residual vector of $y = 2.75x + 2.25$ is $\overline{\vphantom{b}r}_2$ . Each is the target vector minus the model’s predictions, such as $2.9(1) + 1.9 = 4.8$ and $2.75(2) + 2.25 = 7.75$ :

\overline{\vphantom{b}r}_1 = \begin{pmatrix*}[r] 5 \\ 7 \\ 11 \\ 14 \\ 16 \end{pmatrix*} - \begin{pmatrix*}[r] 4.8 \\ 7.7 \\ 10.6 \\ 13.5 \\ 16.4 \end{pmatrix*} = \begin{pmatrix*}[r] 0.2 \\ -0.7 \\ 0.4 \\ 0.5 \\ -0.4 \end{pmatrix*}, \qquad\qquad \overline{\vphantom{b}r}_2 = \begin{pmatrix*}[r] 5 \\ 7 \\ 11 \\ 14 \\ 16 \end{pmatrix*} - \begin{pmatrix*}[r] 5 \\ 7.75 \\ 10.5 \\ 13.25 \\ 16 \end{pmatrix*} = \begin{pmatrix*}[r] 0 \\ -0.75 \\ 0.5 \\ 0.75 \\ 0 \end{pmatrix*},

With $n = 5$ , the lengths and RMSEs are

\begin{aligned} \|\overline{\vphantom{b}r}_1\| &= \sqrt{0.2^2 + (-0.7)^2 + 0.4^2 + 0.5^2 + (-0.4)^2} = \sqrt{0.04 + 0.49 + 0.16 + 0.25 + 0.16} = \sqrt{1.10} \approx 1.049,\\ \|\overline{\vphantom{b}r}_2\| &= \sqrt{0^2 + (-0.75)^2 + 0.5^2 + 0.75^2 + 0^2} = \sqrt{0 + 0.5625 + 0.25 + 0.5625 + 0} = \sqrt{1.375} \approx 1.173, \end{aligned}

\text{RMSE}_1 = \frac{\sqrt{1.10}}{\sqrt{5}} = \sqrt{0.22} \approx 0.469, \qquad \text{RMSE}_2 = \frac{\sqrt{1.375}}{\sqrt{5}} = \sqrt{0.275} \approx 0.524.

The least squares model fits better, because $1.049 < 1.173$ and $0.469 < 0.524$ .

Example 5

The MMAC dataset MonthlyUnemployment has the least squares model

\text{Rate} = -0.07053626\,\text{Months} + 10.17802260.

Compare its RMSE with the RMSE of the line through the first and last data points, $(1, 9.8)$ and $(60, 5.6)$ .

ShowHide solution

Compute the RMSE of each model in R with $n = 60$ : the least squares fit stored from project(), and the line through the first and last points from the slope formula.

cf = project(Rate ~ Months + 1, data = MonthlyUnemployment)
b = cf[1]
m = cf[2]
f = makeFun(m*Months + b ~ Months)
Rate = MonthlyUnemployment$Rate
Months = MonthlyUnemployment$Months
r1 = Rate - f(Months)
sqrt(dot(r1, r1))
[1] 1.50521
sqrt(dot(r1, r1))/sqrt(60)
[1] 0.1943218
m2 = (5.6 - 9.8)/(60 - 1)
b2 = 9.8 - m2*1
g = makeFun(m2*Months + b2 ~ Months)
r2 = Rate - g(Months)
sqrt(dot(r2, r2))
[1] 2.945494
sqrt(dot(r2, r2))/sqrt(60)
[1] 0.3802616

The least squares model fits better, because its RMSE of 0.194 is smaller than the RMSE of 0.380 for the line through the first and last points.

Example 6

Linear models A, B, and C describe the same dataset. Model A has residual vector

\overline{\vphantom{b}r}_A = \begin{pmatrix*}[r] 0.4 \\ -0.6 \\ 0.2 \\ 0.5 \\ -0.3 \end{pmatrix*}.

Model B has $\|\overline{\vphantom{b}r}_B\| = 1.2$ . Model C has $\text{RMSE}_C = 0.40$ . Determine which model fits best and which fits worst.

ShowHide solution

The five components of $\overline{\vphantom{b}r}_A$ give $n = 5$ data points. Convert Model A’s residual vector and Model B’s residual length to RMSEs:

\text{RMSE}_A = \frac{\sqrt{0.16 + 0.36 + 0.04 + 0.25 + 0.09}}{\sqrt{5}} = \frac{\sqrt{0.90}}{\sqrt{5}} = \sqrt{0.18} \approx 0.424,

\text{RMSE}_B = \frac{1.2}{\sqrt{5}} \approx 0.537, \qquad \text{RMSE}_C = 0.400.

Model C fits best and Model B fits worst, because $0.400 < 0.424 < 0.537$ .

Lesson map