Estimating f
Every model in this course is an attempt to guess a function you will never actually see. Six ideas that decide what kind of guess you are making.
Dr. Asaf Madar · lecturer
What is machine learning?
In Machine Learning, we are interested in estimating the function that maps between inputs X and an output y.
X could be cortical thickness, and y could be age. Firing rate to stimulus contrast. Voxel patterns to what a person is looking at.
We call this connecting function f:
x here is list of measurements of one feature (or "predictor").
y is another list of meausrements, or labels, which represent the outcome (or "target").
ε is everything that we cannot, or did not measure and control for: measurement noise, biological variability, the variables we did not record.
ε is why two subjects with identical x still give you different y, and why two measurements from the same subject still give you different y.
Importantly, we never get to see f.
We only see a list of noisy (x, y) pairs. Machine learning is the process of using that list to build an estimation of f, called f (f-hat), that is good enough to be useful.
For example, let's imagine a neuron in auditory cortex that fires to the sounds of pure tones, and the function f that maps the two is a fifth-degree polynomial.
The simulation knows that polynomial, but we just get a table of observations:
Plotted, those rows become a scatter.
Try to draw the curve yourself first, then switch on the estimate, and only then reveal the truth.
This cell is not a textbook bell. It has a best frequency near 7 kHz, a suppressed band around 15 kHz, and a second response region near 23 kHz. Not a shape we could guess from a handful of trials.
Root-mean-square gap between the two curves. More trials, or less noise, pulls it down. It never reaches zero: a smooth estimate can only chase so many bends.
Many different curves are compatible with our data. Adding data does not change the underlying f, but it narrows down which f can be estimated.
Fitting vs. predicting
In fitting, we have all our data, we fit a model to it, and we interpret the coefficients:
Does salary rise with years of education, and by how much? Does accuracy drop as memory load increases?
The model describes the data in front of us, and the coefficient is the answer. This is what psychology and economics usually ask for.
In predicting, we fit on one part of the data, called the training set, then ask the model for y on points it has never seen, called the test set.
Here it does not matter how well the model described the training data. What matters is whether it works on the next subject.
The two come apart quickly. A flexible enough model can pass through every training point exactly, so its training error goes to zero. It has learned the noise.
how well it fits—
how well it predicts—
Degree 1 is a straight line. Degree 12 is a curve with twelve bends, and it can be dragged through almost anything.
Training error always improves as we add flexibility. Test error goes down, bottoms out, then climbs back up. That U-shape is the bias-variance tradeoff, and finding its bottom is most of applied ML.
Often we want both: estimate f on a training set, check on a test set that it really predicts, then interpret the coefficients to see which features carried the signal.
The third step is only honest if the second one passed.
Supervised vs. unsupervised
The dividing question is simple: do we have y?
If every observation arrives with a label (this scan is from a patient, this trial was a face, this subject is 34 years old), we are doing supervised learning.
Someone already decided what the right answer is, so we can measure how often we get it wrong, and then minimise that.
What we have. Colour is the diagnosis, and it was given to us. No algorithm worked it out.
What the method produces. A boundary. Any new subject falls on one side and gets that label, and because we know the truth, every mistake can be counted.
If there is no y, we are doing unsupervised learning: looking for structure that is already in x.
Which brain regions behave alike? How many cell types are in this recording?
Nothing here is right or wrong, so there is no error to minimise.
Clustering is the usual first step. We hand over the points and one number, k, the number of groups to look for, and the algorithm splits the data so that points in a group sit close together.
It repeats two steps until nothing moves: assign every point to its nearest centre, then recompute each centre from the points assigned to it.
The data below really has three groups, but the algorithm is never told that. Ask it for the wrong number and see what happens:
What we have. 150 recordings, two measurements each. No colour, because there is no label to colour by.
Regression vs. classification
Within supervised learning, the split is about the type of y:
Continuous y gives us regression. Categorical y gives us classification.
That choice decides which model we reach for, and which number we report.
The average squared miss. Squaring means one wild error costs far more than several small ones. It also means the units are y squared, which is why we often report the square root instead.
Every square is a subject. This “model” predicts healthy for all of them, always.
Accuracy climbs toward 99% while the model catches exactly zero patients. Always report a measure that can see the class we actually care about.
Look before you model
Exploratory data analysis is not a warm-up. It is where we find out that a sensor was disconnected for subject 12, that age and scanner site are confounded, or that our “continuous” variable is really four discrete values.
Two habits are worth building now.
Know what a correlation looks like
Neuroscience papers are full of r = 0.3. Most of us picture a much tighter cloud than that. Calibrate your eye:
At r = 0.3 the model accounts for 9% of the variance in y. The cloud looks close to round, and it should.
Never trust a summary statistic on its own
These four datasets share the same mean x, the same mean y, the same variances, the same correlation, and the same fitted line, to two decimal places.
Anscombe built them in 1973 to make exactly this point, and it has not aged a day.
I: a real linear relationship. II: a curve, badly served by a line. III: a perfect line plus one outlier that tilts it. IV: no relationship at all, plus one point with total leverage.
Plot the data. Then plot the residuals. Two summary numbers and a p-value cannot tell any of these four apart.
The linear model, least squares, the OLS derivation, R², and what small samples do to all of it.