17.3.1. Estimating the Coefficients - Ordinary Least Squares (OLS)#

To estimate the coefficients of our optimal model, we should first define what is a good model. We will say that a good model is one that predicts well the $Y$ variable from the $X$ one. We already know from the example above that, since the relationship is not perfectly linear, the model will make some mistakes.

Let $\{(x_i,y_i)\}$ be our set of observations. Let

be the prediction of the model for the observation $x_i$. For each data point $(x_i,y_i)$, we will define the deviation of the prediction from the $y_i$ value as follows:

These numbers will be positive or negative based on whether we underestimate or overestimate the $y_i$ values. As a global error indicator for the model, given the data, we will define the residual sum of squares (RSS) as:

or equivalently:

This number will be the sum of the square values of the dashed segments in the plot below:

Intuitively, if we minimize these numbers, we will find the line which best fits the data.

We can obtain estimates for ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ by minimizing the RSS using an approach called ordinary least squares.

We can write the RSS as a function of the parameters to estimate:

This is also called a cost function or loss function.

We aim to find:

The minimum can be found setting:

Doing the math, it can be shown that:

17.3.2. Interpretation of the Coefficients of Linear Regression#

Using the formulas above, we find the following values for the example above:

beta_0: 39.94
beta_1: -0.16

These parameters identify the following line:

The plot below shows the line on the data:

Apart from the geometric interpretation, the coefficients of a linear regressor have an important statistical interpretation. In particular:

The intercept ${\beta }_0$ is the value of $y$ that we get when the input value $x$ is equal to zero $x=0$ (i.e., $f(0)$). This value may not always make sense. For instance, in the example above, we have: ${\beta }_0=39.94$, which means that, when the horsepower is $0$, then the consumption in mpg is equal to $39.94$.

The coefficient ${\beta }_1$ indicates the steepness of the curve. If ${\beta }_1$ is large, then the curve is steep. This indicates that a small change in $x$ is associates to a large change in $y$. In general, we can see that:

which reveals that when we observe an increment of one unit of x, we observe an increment of ${\beta }_1$ units in y. In our example, ${\beta }_1=-0.15$, hence we can say that, for cars with one additional unit of horsepower, we observe an drop in mps $-0.15$ units.

17.3.3. Accuracy of the Coefficient Estimates#

Recall that we are trying to model the relationship between two random variables $X$ as $Y$ with a simple linear model:

This means that, once we find appropriate values of ${\beta }_0$ and ${\beta }_1$, we expect these to summarize the linear relationship in the population or the population regression line. Also, recall that these values are obtained using two formulas which are based on realizations of $X$ of $Y$ and can be hence seen as estimators:

We now recall that, being estimates, they provide values related to a given realization of the random variables.

Let us consider an ideal population for which:

Ideally, given a sample from the population, we expect to obtain ${\hat{\beta }}_0\approx 1$ and ${\hat{\beta }}_1\approx 2$. In practice, different samples may lead to different estimates and hence different regression lines, as shown in the plot below:

Each of the first three subplots shows a different sample drawn from the population, with its corresponding estimated regression line, along with the true population regression line. The last subplot compares the different estimated lines with the population regression line and the average regression line (in red). Coefficient estimates are shown in the subplot titles.

As can be noted, each estimate can be inaccurate, while the average regression line is very close to the population regression line. This is due to the fact that our estimators for the parameters of the regression coefficients have non-zero variance. In practice, in can be shown that these estimators are unbiased (hence the average regression line is close to the population line).

17.3.3.2. Confidence Intervals of the Regression Coefficients#

As we have seen, standard errors allow to compute confidence intervals. We will not see it into details, but it turns out that the true values will be in the following intervals with a $95\mathrm{\%}$ confidence:

$$[{\hat{\beta }}_0-1.96\cdot SE({\hat{\beta }}_0),{\hat{\beta }}_0+1.96\cdot SE({\hat{\beta }}_0)]$$

$$[{\hat{\beta }}_1-1.96\cdot SE({\hat{\beta }}_1),{\hat{\beta }}_1+1.96\cdot SE({\hat{\beta }}_1)]$$

In practice, it is common to compute confidence intervals with a $95\mathrm{\%}$ confidence. In the case of our model:

$$horsepower\approx ={\beta }_0+mpg\cdot bet{a}_1$$

We will have:

	COEFFICIENT	STD ERROR	CONFIDENCE INTERVAL
${\beta }_0$	39.94	0.717	$[38.53,41.35]$
${\beta }_1$	-0.1578	0.006	$[-0.17,-0.15]$

From the table above, we can say that:

• The value of mpg for horsepower=0 lies somewhere between $38.53$ and $41.35$;

• An increase of horsepower by one unit is associated to an decrease of mpg between $-0.17$ and $.015$.

It is also common to see plots like the following one:

In the plot above, the shading around the line illustrates the variability induced by the confidence intervals estimated for the coefficients.

17.3.4. Accuracy of the Model#

The tests performed above will tell us if there is a relationship between the variables, but they will not tell us how well does the model fit the data. For instance, if the relationship between $X$ and $Y$ is not linear, we would expect the model not to fit the data very well. In practice, we can use different measures of accuracy of the model.

17.3.4.2. $R^2$ Statistic#

The RSE value is an absolute measure, which is measured in the units of $Y$. Indeed, to interpret it, we had to check the range of the $Y$ values. An alternative way to check if the model is fitting the data well, would be to compare the performance of our model with the performance of a baseline model which assumes no association between the $X$ and $Y$ variables. This model would be:

$$Y=k$$

This model has a single parameter $k$. The Residual Sum of Squares of this model would be:

$$RSS(k)=\sum _{i=1}^n(k-y_i{)}^2$$

To find the optimal value $k$, we can compute the derivative of the RSS and set it to zero:

$$\frac{\partial RSS(k)}{\partial k}=2\sum _{i=1}^n(k-y_i)=2(nk-\sum _{i=1}^n{y}_i)=2(nk-n\bar{y})$$

$$2n(k-{\bar{y}}_i)=0\iff k={\bar{y}}_i$$

Hence, the optimal estimator, when there is not relationship between $X$ and $Y$ is the average value of $Y$. We will call its RSS value the total sum of squares:

$$TSS=\sum _{i=1}^n(y_i-\bar{y}{)}^2$$

We can compare the RSS value obtained by our model to the TSS, which is the error of the baseline method:

$$\frac{RSS}{TSS}$$

This number will be comprised between 0 and 1, and in particular:

• $\frac{RSS}{TSS}=0$ when $RSS=0$, i.e., we have a perfect model;

• $\frac{RSS}{TSS}=1$ when $RSS=TSS$, i.e., we are not doing any better than the baseline model (so our model is poor).

Note that the RSS measures the variability in $Y$ left unexplained after regression (the one that the model could not capture), while the TSS measures the total variability in $Y$. The fraction hence explains the proportion of variability which the model could not explain.

We define the $R^2$ statistic as:

$$R^2=1-\frac{RSS}{TSS}$$

Inverting by the $1-$ subtraction, this number measures the proportion of variability in $Y$ that can be explained using $X$

The plot below shows examples of linear regression fits with different $R^2$ values.

The interpretation of the $R^2$ is related to the one of the Pearson’s $\rho$ correlation coefficient. Indeed, both scores are independent of the slope of the regression line, but quantify how liner the relationship between $X$ and $Y$ is. In practice, it can be shown that:

$$R^2={\rho }^2$$

The $R^2$ value in our example of regressing mpg from horsepower is:

$$R^2=0.61$$

Which indicates that $61\mathrm{\%}$ of the variance of the $Y$ variable is explained by the model.

17.4.1. Geometrical Interpretation#

The multiple regression model based on two variable has a geometrical interpretation. Indeed, the equation:

identifies a plane in the 3D space. We can visualize the plane identified by the $mpg={\beta }_0+{\beta }_{1}horsepower+{\beta }_{2}weight$ model as follows:

The dashed lines indicate the residuals. The best fit of the model minimizes again the sum of squared residuals. This model makes predictions selecting the $Z$ value which intersect the plane for given values of $X$ and $Y$.

In general, when we consider $n$ variables, the linear regressor will be a (n-1)-dimensional hyperplane in the n-dimensional space.

17.4.2. Statistical Interpretation#

The statistical interpretation of a multiple linear regression model is very similar to the interpretation of a simple linear regression model. Given the general model:

we can interpret the coefficients as follows:

• The value of ${\beta }_0$ indicates the value of $y$ when all independent variables are set to zero;

• The value of ${\beta }_i$ indicates the increment of $y$ that we expect to see when $x_i$ increments by one unit, provided that all other values $x_j|j\ne i$ are constant.

In the considered example:

we obtain the following estimates for the coefficients:

We can interpret these estimates as follows:

• Cars with zero horsepower and zero weight will have an mpg of $45.64$ ($\approx 19.4Km/l$).

• An increment of one unit of horsepower is associated to a decrement of mpg of $-0.05$ units, provided that weight is constant. This makes sense: cars with more horsepower will probably consume more fuel.

• An increment of one unit of weight is associated to a decrement of mpg of -0.01 units, provided that horsepower is constant. This makes sense: heavier cars will consume more fuel.

Let’s compare the estimates above with the estimates of our previous model:

In that case, we obtained:

We can note that the coefficients are different. This happens because, when we add more variables, the model explains variance in a different way. If we think more about it, this is coherent with the interpretation of the coefficients. Indeed:

• $39.94$ is the expected value of mpg when horsepower=0, but all other variables have unknown values. $45.64$ is the expected value of mpg when horsepower=0 and weight=0. This is different, as in the second case we are (virtually) looking at a subset of data for which both horsepower and weight are zero, while in the first case, we are only looking at data for which horsepower=0, but weight can be any value. In some sense, we can see $39.94$ as an average value for different values of weight (and all other unobserved variables).

• $-0.16$ is the expected increment of mpg when we observe an increment of one unit of horsepower and we don’t know anything about the values of the other variables. $-0.05$ is the expected increment of mpg when horsepower and weight are held constant, so, again, we are (virtually) looking at a different subset of the data in which the relationship between mpg and horsepower may be a bit different.

Note that, also in the case of multiple regression, we can estimate confidence intervals and perform statistical tests. In our example, we will get this table:

17.4.3. Estimating the Regression Coefficients#

Given the general model:

We can define our cost function again as the residual sum of squares:

Where $m$ is the total number of observations and $x_j^{(i)}$ is the $j^{th}$ variable ($x_j$) of the $i^{th}$ observations.

The values ${\hat{\beta }}_0,\dots ,{\hat{\beta }}_n$ values that minimize the loss function above are the multiple least square coefficient estimates.

To find these optimal values, it is convenient to use matrix notation. Given $m$ observations, we will have $m$ equations:

We can write the $m$ equations in matrix form as follows:

where:

The matrix $\mathbf{X}$ is called the design matrix.

In the notation above, we want to minimize:

It can be shown that, by the least squares method, the RSS is minimized by the estimate:

17.4.4. The F-Test#

When fitting a multiple linear regressor, it is common to perform a statistical test to check whether at least one of the regression coefficients is significantly different from zero (in the population). This test is called an $F-test$. We define the null and alternative hypotheses as follows:

The null hypothesis (the one we want to reject with this test) is that all coefficients are zero in the population. If this is true, than the multiple regressor is not reliable and we should discard it. The alternative hypothesis is that at least one of the coefficients is different from zero.

The test is performed by computing the following F-statistic:

Where recall that $n$ is the number of variables and $m$ is the number of observations.

In practice, the F-statistic will be:

• Close to $1$ if there is no relationship between the response and the predictors ($H_0$ is true);

• Greater than $1$ if $H_a$ is true.

The test is carried out as usual, finding a p-value which indicates the probability to observe a statistic larger than the observed one if all regression coefficients are zero in the population.

In our example of regressing mpg from horsepower and weight, we will find:

This indicates that the regressor is statistically relevant. The $F-statistic$ is much larger than $1$ and the p-value (Prob(F-statistic)) is very small (under the significance level $\alpha =0.05$).

17.4.5. Variable Selection#

Let’s now try to fit a multiple linear regressor on our dataset by including all variables. Our dependent variable will be mpg, while the set of dependent variables will be:

['displacement' 'cylinders' 'horsepower' 'weight' 'acceleration'
 'model_year' 'origin']

We obtain the following measures of fit:

The regressor has a good $R^2$ and the p-value of the F-test is very small. We can conclude that there is some relationship between the independent variables and the dependent one.

The estimates of the regression coefficients will be:

From the table, we can see that not all predictors have a p-value below the significance level $\alpha =0.05$. In particular:

• horsepower has a large p-value of $0.22$;

• cylinders has a large p-value of $0.128$;

• acceleration has a large p-value of $0.415$.

This means that, within the current regressor, there is no meaningful relationship between these variables and mpg. A legitimate question is

How is it possible that horsepower is not associated to mpg in this regressor if it was associated to it before?!

However, we should recall that, when we consider a different set of variables, the interpretation of the coefficients changes. So, even if in the previous models, horsepower was correlated to mpg, now it is not correlated anymore. We can imagine that the relationship between these variables is now explained by the other variables which we have introduced.

Even if the model is statistically significant, it does make sense to get rid of the variables with poor relationships with mpg. After all, if we remove a variable, the estimates of the other coefficients may change.

A common way to remove these variables is by backward selection or backward elimination. This consists in iteratively removing the variable with the largest p-value. We remove one variable at a time and re-compute the results, iterating until all variables have a small p-value.

let’s start by removing acceleration. This is the result:

Note that the coefficients have changed. We now remove cylinders, which has the largest p-value of $0.117$:

All variables now have an acceptable p-value ($\alpha =0.05$). We are done. Note that, by removing the two variables, horsepower now has an acceptable p-value. This indicates that one of the removed variables was redundant with respect to horsepower.

17.4.6. Adjusted $R^2$#

While in the case of simple regression we saw that $R^2=\rho (x,y{)}^2$ (where $\rho$ is the correlation coefficient), in the case of multiple regression, it turns out that:

In general, having more variables in the linear regressor will reduce the error term and improve the covariance between $Y$ and $\hat{Y}$, hence increasing the $R^2$. However, in general having a small increase in $R^2$ when we add a new variable may not be good. Indeed, we could prefer a simpler model with a slightly smaller $R^2$ value.

To express this, we can compute the adjusted $R^2$ as follows:

Where $m$ is the number of data points and $n$ is the number of independent variables. The ${\bar{R}}^2$ re-balances the $R^2$ accounting for the introduction of additional variables.

For instance the last model we fit has an $R^2=0.820$ and ${\bar{R}}^2=0.818$. We will see more in details how to use the ${\bar{R}}^2$ in the laboratory.

17.5.1. Predictors with Only Two Levels#

We will first see the case in which qualitative predictors only have two levels. To handle these as independent variables, we can define a new dummy variable which will encode $1$ as one of the two levels and $0$ as the other one. For instance, we can introduce a fueltype[T.gas] variable defined as follows:

If we fit the model:

We obtain an $R^2=0.661$ with $Prob(F-statistic)\approx 0$ and the following estimates for the regression parameters:

How do we interpret this result?

• The value of mpg when horsepower=0 fueltype=diesel (i.e., fueltype[T.gas]=0) is $41.2379$;

• An increase of one unit of horsepower is associated to a decrease of $0.1295$ units of mpg provided that fueltype=diesel;

• For gas vehicles we expect to see a decrease of mpg equal to $2.7658$ with respect to diesel vehicles.

17.5.2. Predictors with More than Two Levels#

When predictors have $n$ levels, we need to introduce multiple dummy variables. Specifically, we need to introduce $n-1$ binary variables. For instance, if the levels of the variable income are low, medium and high, we could introduce two variables income[T.low] and income[T.medium]. These are sufficient to express all possible values of income as shown in the table below:

Note that we could have introduced a new variable income[T.high] but this would have been redundant and so correlated to the other two variables, which is something we know we have to avoid in linear regression.

If we fit the model which predicts mpg from horsepower and fuelsystem we obtain $R^2=0.734$, $Prob(F-statistic)\approx 0$ and the following estimates for the regression coefficients:

As we can see, we have added different correlation coefficients in order to deal with the different levels. Not all predictors have a low p-value, so we can remove those with backward elimination. We will see some more examples in the laboratory.

17.6.1. Interaction Terms#

A linear regression model assumes that the effect of the different independent variables to the prediction of the dependent variable is additive:

In some cases, however, it makes sense to assume the presence of interaction terms, i.e., terms in the model which account for the interactions between some variables. For instance:

As a concrete example, consider the problem of regressing mpg from horsepower and weight. A simple model of the kind:

assumes that the ${\beta }_1$, the increment that we observe in mpg when horsepower increments by one unit is constant even when other variables change their values. However, we can imagine how, for light vehicles horsepower affects mpg in a way, while for heavy vehicles horsepower affects mpg in a different way. For example, we expect light vehicles with big horsepower to be more efficient than heavy vehicles with small horsepower. To account for this, we could consider the following model instead:

Note that the model above is nonlinear in $horsepower$ and $weight$, but it is still linear if we introduce a variable $hw=horsepower\times weight$. We can easily do this by adding a new column to our design matrix computed as the product of the two variables. Then we can fit the model using the same exact estimators seen before.

This new regressor obtained $R^2=0.748$, which is larger as compared to $R^2=0.706$ obtained for the $mpg={\beta }_0+{\beta }_{1}horsepower+{\beta }_{2}weight$ regressor. Both regressors have a large F-statistic and an associated p-value close to zero. The new regressor explains more variance than the previous one.

The estimated coefficients are:

Let’s compare these with the ones obtained for the $mpg={\beta }_0+{\beta }_{1}horsepower+{\beta }_{2}weight$ regressor:

We can note that:

• The p-value of the product $horsepower\times weight$ is almost zero. This is a strong evidence that the true relationship is not merely additive, but an interaction between the two variables actually happens.

• The coefficients of horsepower and weight have changed. This makes sense, also their interpretation has changed. For instance, in the new regressor an increase of one unit of horsepower is associated to a decrease of $0.2508$ units of mpg if the product between horsepower and weight does not change, i.e., if the way the two quantities interact does not change.

How should we interpret the coefficient ${\beta }_3$? Let’s rewrite our model:

as follows:

Hence we can see ${\beta }_3$ as the increase in the coefficient of horsepower when weight increases by one unit. In the example above ${\beta }_3$ is very small, this is due to the fact that weight is measured in pounds, so increasing the weight by one pound has a very small effect. It easy to see that, an increase in weight by $1000$ pounds increases the coefficient of $horsepower$ by $1000{\beta }_3\approx =0.05$. This is a relevant increase, considering that the coefficient of $horsepower$ is $-0.2508$. Hence, when we see an increase in weight of $1000$ units, the coefficient of horsepower will be larger, meaning that in heavier cars, large horsepower will let mpg decrease more gently.

Let us consider the computed coefficients again:

We can interpret the results as follows:

• Intercept: the value of mpg is $63.5579$ when horsepower=0 and weight=0 (also their product will be zero);

• horsepower: an increase of one unit corresponds to a decrement of $-0.2508$ in mpg provided that weight is held constant and the way weight influences horsepower does not change. Note that, if we change the value of horsepower and keep weight constant, the interaction term will change. Here we assume that, even if the ratio change, we assume that the way weight affects horsepower does not change.

• weight: an increase of one unit corresponds to a decrement of $-0.0108$ in mpg provided that horsepower is held constant and the way horsepower affects weight does not change.

• horsepower:weight: an increase in one unit of weight increases the coefficient regulating the effect of horsepower on mpg by $5.05e-5$.

17.6.2. Non-linear Relationships: Quadratic and Polynomial Regression#

In many cases, the relationship between two variables is not linear. Let’s visualize the scatterplot between $X$ and $Y$:

This relationship does not look linear. It looks instead as a quadratic, which would be fit by the following model:

Again, the model above is nonlinear in horsepower, but we can still fit it with a linear regressor if we add a new variable $z=horsepower$.

The fit model will obtain $R^2=0.688$, larger than $R^2=608$ obtained by the base model ($mpg={\beta }_0+{\beta }_{1}horsepower$). Both have a large F-statistic.

The estimated coefficients are:

The coefficients now describe the quadratic:

If we plot it on the data, we obtain the following:

In general, we can fit a polynomial model to the data, choosing a suitable degree $d$. For instance, for $d=4$ we have:

which identifies the following fit:

This approach is known as polynomial regression and allows to turn the linear regression into a nonlinear model. Note that, when we have more variables, polynomial regression also includes interaction terms. For instance, the linear model:

becomes the following polynomial model of degree $2$:

As usual, we only have to add new variables for the squared and interaction term and solve the problem as a linear regression one. This is easily handled by libraries. Note that, as the number of variables increases, the number of terms to add for a given degree also increases.

17.8.1. Ridge Regression#

One way to remove collinearity is to remove those variables which are highly correlated with other variables. This will effectively remove collinearity and solve the instability issues of linear regression. However, when there is a large number of predictors, this approach may not be very convenient. Also, it is not always clear which predictors to remove and which ones to keep, especially in the case of multicollinearity (sets of more than two variables which are highly correlated).

A “soft approach” would be to feed all variables to the model and encourage it to set some of the coefficients to zero. If we could do this, the model would effectively select which variables to exclude from the model, hence releasing us from making a hard decision.

This can be done by changing our cost function, the RSS. Recall that the RSS of linear regression (Ordinary Least Squares) can be written as:

where $x_{ij}$ is the value of the $x_j$ variable in the $i^{th}$ observation.

We can write the cost function as follows:

In practice, we are adding the following term:

$\sum _{j=1}^n{\beta }_j^2$

weighted by a parameter $\lambda$. The term above penalizes solutions with large values for the coefficients ${\beta }_j$, hence encouraging the model to put some of the variables to zero, or close to zero. This term is called “regularization term” and a linear regressor fit by minimizing this new cost function is called a ridge regressor.

The cost function has a parameter $\lambda$ which must be set as a constant. These kinds of parameters which we need to specify manually and are not computed from the data, are called hyperparameters. In practice, $\lambda$ controls how much we want to regularize the linear regressor. The higher the value of $\lambda$, the stronger the regularization will be, hence encouraging the model to select small values of ${\beta }_j$ or set some of them to zero.

To avoid $\lambda$ having different effects depending on the unit measures of the given variables, it is common to z-score all variables before computing a ridge regressor.

This is shown in the following plot, which shows the values of the parameters ${\beta }_j$ when a ridge regression model is fit with different values of $\lambda$:

As can be seen, the coefficient values are gradually shrunk towards zero as the value of $\lambda$ increases. Interestingly, the model “decides” which coefficients to shrink. For instance, for low values of $\lambda$, weight is shrunk, while acceleration is first shrunk, then allowed to be larger than zero. This is due to the fact that, the regularization term acts as a sort of “soft constraint” encouraging the model to find smaller weights, while still finding a good solution.

It can be shown (but we will not see it formally), that ridge regression reduces the variance of coefficient estimates. At the same time, the bias is increased. So, finding a good value of $\lambda$ allows to control the bias-variance trade-off.

17.8.2. Lasso Regression#

The ridge regressor will in general set coefficients exactly to zero only for very large values of $\lambda$. An alternative model which has been shown to set coefficients more likely to zero is the lasso regressor. The main difference with ridge regression is in the regularization term. The new cost function of a lasso regressor is as follows:

We basically replaced the norm of the coefficients with the absolute values (this is called an L1 norm), which has the effect to encourage the model to set values to zero.

The figure below shows how the coefficient estimates change for different values of $\lambda$:

As can be seen, the model now makes “hard choices” on whether a value should be set to zero or not, thus performing variable selection. Once Lasso regression identified the variables to remove, we could create a reduced dataset keeping only columns with nonzero coefficients and fit a regular, interpretable, linear regressor with the original, non standardized, data.