The Price of
Flexibility
Four ways of letting a model bend, and the bill that arrives with each one. Interactions, nearest neighbours, and the trade that decides how much bending is worth paying for.
Dr. Asaf Madar · lecturer
Interaction with a group
Last class every predictor was a number. Plenty are not: sex, diagnosis, scanner site, task condition.
A two-category variable becomes a dummy: one column that is 0 or 1.
With k categories we need k − 1 dummies, never k. Three sites take two columns, and the category left out becomes the baseline: the one β0 describes. Adding the third column would let the model write the same prediction in infinitely many ways, and there would be no unique answer to solve for.
Now put the dummy next to a continuous predictor. There are three models we could write, and the difference between them is the point of this tab:
Read model 3 by substituting the two values of d:
- Controls, d = 0: y = β0 + β1x
- Patients, d = 1: y = (β0 + β2) + (β1 + β3)x
Two separate lines. β2 shifts one up or down, and β3 tilts it. That tilt is the interaction: the effect of age on memory is not the same in the two groups.
Set the slope difference to zero and model 3 stops earning its extra parameter: the two lines come out parallel, and β3 is just noise.
Without the interaction term, the model is forced to give both groups the same slope. It can move the lines apart but never let them converge or cross. If the groups really do differ in slope, that constraint shows up as a systematic pattern in the residuals.
Two continuous predictors
Nothing about an interaction needs a category. Multiply two continuous predictors together and the same thing happens, only now the moderator varies smoothly instead of switching between two values.
The trick from the previous tab still works. Collect the x1 terms together:
Read it as a line in x1 whose slope is itself a straight-line function of x2.
Every value of x2 gives a different line, so instead of two lines we get a whole family of them. With β3 = 0 the family is parallel. Turn β3 up and it fans out.
Below, sleep predicts a memory score, and age is the moderator: an interaction means a night of sleep is worth more to some ages than to others.
Each line is the model evaluated at one age, holding age fixed and sweeping sleep. The slope of each line is β1 + β3x2, printed beside it.
The same model as a surface. Without the interaction it is the flat plane from last class. With it, the plane acquires a twist: a saddle, high at two opposite corners and low at the other two.
Straight along either axis, curved along the diagonals. Every line you can draw on it parallel to an axis is still straight, which is why this is still a linear model.
“Linear” has always meant linear in the coefficients, not in the predictors. x1x2 is just another column in X, so least squares solves it with exactly the same normal equations.
An interaction is not a new kind of model. It is one more column, built by multiplying two columns we already had. Everything from last class still applies: the same RSS, the same derivation, the same solution.
K-nearest-neighbours regression
Every model so far assumed a shape and then estimated its coefficients. KNN assumes nothing at all.
To predict at a point x0: find the K training points closest to it, and average their y.
That is the whole method. There is no training step and nothing to solve: the model is the data. Methods like this are called non-parametric, because the number of things they remember grows with n instead of being fixed at p + 1.
K controls how much smoothing happens. K = 1 copies the nearest neighbour exactly, noise and all. Large K averages over a wide neighbourhood and flattens toward the overall mean.
When the truth really is a line, linear regression wins and no value of K catches it: the parametric model already knows the answer's shape, so it spends all its data estimating two numbers. When the truth curves, that assumption becomes a liability and KNN overtakes it.
Why this stops working in neuroscience
KNN needs neighbours that are genuinely near. In one dimension that is easy. As p grows, the space empties out: to capture 10% of the data in a cube, you need to reach 10% of the range along one axis, but 63% along each of ten axes. The “nearest” neighbours are no longer near, and the average is taken over points that have little to do with x0.
The side length of the cube you need in order to enclose a fixed fraction of a uniformly filled space, as dimensions are added.
A 500-voxel pattern is 500 dimensions. This is why a flexible method that looks unbeatable on one predictor can be worse than a straight line on real imaging data.
Bias and variance
Last class we watched test error fall, bottom out, then climb as a model got more flexible. This tab is the reason it does that.
Imagine repeating the whole study many times: same population, same design, a fresh cohort each time. Every cohort gives a slightly different fitted model. Two separate things can go wrong.
- Bias is how far the average of those fits sits from the truth. It is the error the model makes by being the wrong shape, and more data will not remove it.
- Variance is how much the fits differ from each other. It is the error that comes from having fitted this particular cohort rather than another.
Below, the study is repeated thirty times at whatever flexibility you choose.
Squared bias and variance are averaged across the range of x. Their sum, plus the noise that no model can remove, is the expected test error at a new point.
Three terms, and only the first two are ours to trade. The third is the irreducible noise from class 1: the part of y that no model, however good, can recover.
The two we control move in opposite directions: flexibility buys lower bias at the price of higher variance. The best model is not the one with the least bias, nor the one with the least variance, but the one whose sum is smallest.
Squared bias falls, variance rises, and their sum is U-shaped. Where that U bottoms out is the flexibility worth choosing. The vertical axis is logarithmic, because the two curves span two orders of magnitude across this range.
Drag the subject count in the panel above and watch the curve shift. More data lowers variance at every degree, which nudges the sweet spot rightward: bigger samples can afford more flexibility.
Past that point the measured bias drifts upward too. With only thirty cohorts the average fit is itself an estimate, and once variance is large that estimate gets noisy. Real bias does not rise; our measurement of it does.
A model that is too simple is wrong the same way every time. A model that is too flexible is wrong a different way every time. Both cost you, and the cost is measured on data the model has not seen.
Glossary
Everything introduced today, on top of the class 1 glossary.
tradeoff
Next time
What to do when a model has more flexibility than the data can pay for: shrinking coefficients toward zero, and choosing how much to shrink without touching the test set.
The linear model, RSS, the OLS derivation, R², and the plane for two predictors.