Least Squares

The simplest guess at f is a straight line: we assume y changes by a constant amount for every unit of x.

Two numbers, both readable in plain language:

• β₀, the intercept: the predicted y when x = 0. Here, the recall score of an average-sized hippocampus.
• β₁, the slope: how much y moves per one-unit increase in x. This is the number we interpret.

The slope turns the line about x = 0, marked on the plot. That is the one point it leaves alone, which is why we often measure a predictor from its own mean: it puts the pivot inside the data instead of far outside it.

Fitting the model means nothing more than choosing these two numbers. Try it by hand, you are the optimiser:

We just moved a line until it looked right. To do that automatically, we need to say precisely what “right” means.

For each subject the model predicts y_i = β₀ + β₁x_i, and reality delivered y_i.
The gap between them is the residual, e_i = y_i − y_i: the vertical distance from the point to the line.

We cannot simply add the residuals up, because the positive and negative ones cancel and a terrible line can score zero. We square each one first, then add. That is the residual sum of squares:

Squaring does two jobs: every error counts as positive, and large misses are punished out of proportion. Being off by 4 is sixteen times worse than being off by 1, not four times worse.

Ordinary least squares (OLS) is the choice of β₀ and β₁ that makes RSS as small as it can be.

Below, each residual is a red stick running from the observation to the line. A better line means shorter sticks, all of them at once:

Every pair (β₀, β₁) is one possible line, and each one has its own RSS. Treat that number as a height and the whole space of lines becomes a landscape.

It is a bowl, a paraboloid. One bottom, no local minima to get trapped in, no flat plateaus. That is a rare luxury, and it is why linear regression has an exact formula while most models in this course have to be searched for numerically.

The bowl also sits upright rather than leaning, because x is measured from its own mean. That is the same fact as the pivot in the previous tab: with a centred predictor, the best slope does not depend on the intercept.

On the left is the surface itself. On the right is the same surface seen from directly above, drawn as a contour map. The dot is our current line and the cross is the least-squares solution, on both.

Hunting for the best line by hand is fine for two parameters, but we can do better.
Think of RSS as a function of β. It is a parabola: steep where the line is badly wrong, flattening out as it improves, and perfectly flat at its lowest point.
Flat means the derivative is zero, and that is the whole trick: write RSS as a function of β, differentiate, set it to zero, and solve.

To keep the algebra visible we drop the intercept for now and fit a line through the origin, y = βx, leaving one unknown. Reveal one step at a time:

Putting the intercept back

Doing the same with two unknowns (take the partial derivative with respect to β₀ and to β₁, set both to zero, solve the pair) gives the formulas we actually use:

Same structure as step 05, with every variable measured from its own mean instead of from zero. Two things fall straight out, both worth remembering.

First, β₀ = y − β₁x rearranges to say the fitted line always passes exactly through the point (x, y).
Second, the numerator is the covariance of x and y and the denominator is the variance of x, which means:

RSS on its own is unreadable. Is 4,300 good? It depends on the units of y and on how many subjects we have. Two fixes:

MSE divides RSS by n, so it no longer grows just because we collected more data.
Its square root, RMSE, is back in the original units: “on average we miss by 3.4 points” is a sentence a clinician can use.

R² compares our model to the laziest one available. With no x at all, the best guess for everybody is the mean y.
The error of that null model is the total sum of squares, TSS. R² asks what fraction of it our line removed.

R² = 0 means the line did no better than guessing the mean every time. R² = 1 means the residuals vanished.
Toggle between the two models below and watch the sticks shorten: the fraction of the total we removed is R².

Everything so far works beautifully with 500 clean observations. Neuroscience rarely offers those.
We get 25 subjects, an effect explaining a few percent of the variance, and a hundred candidate predictors.

The demo below is the one to sit with: fit the same linear model on a small training sample, then score it honestly on fresh subjects from the same population.

Two things to notice as you click through cohorts.
At n = 20 the estimated slope swings wildly around the true one. Each cohort tells a different story, and any single published one might be ours.
Train R² is systematically optimistic: it is computed on the very points used to place the line, so it gets to peek at the answer.

Drag n up to 200 and both problems fade. That gap between the two numbers is why this course insists on a test set.

What this looks like in Python

Everything in this class is four lines of scikit-learn. That is exactly the danger: the library will happily let us skip the split.

Nothing so far needed x to be a single measurement. With p features the model just grows a term each:

The recipe does not change: write RSS, take the partial derivative with respect to every β, set them all to zero, and solve the system. With one predictor that was one equation. With p predictors it is p + 1 equations, which a computer solves in microseconds.

What does change is the picture.
One predictor gives a line in a plane. Two predictors give a plane in a cube: for every pair (x₁, x₂) the model returns a height. The residuals are still vertical distances from each point to that surface, and least squares still picks the surface leaving the smallest total.

Below, recall is predicted from hippocampal volume and age, both measured from the group mean.
Three numbers now, so three sliders: one height and two tilts. Drag the figure itself to turn it.

Everything introduced today, in the notation the course will keep using.

Next time

What happens when features correlate with each other, why adding them always helps in-sample and often hurts out-of-sample, and the first tool for dealing with it, regularisation.