The most importat concepts in selecting a statistical learning procedure for a specific data set.

2.2.1 Measuring the Quality of Fit

Evaluating the performance of a statistical learning method on a given data set

= Measurement of how well predictions match observations

MSE (Mean Squared Error)

MSE is the most commonly used measure for regression.

$$ MSE=\frac1n\underset{i=1}{\overset{n}\Sigma}(y_i-\hat f(x_i))^2 $$

where

Train vs Test

We are more interested in test MSE in measuring the quality of the method. However, if no test observations are available(no test data), then one might select a method that minimizes the train MSE.

Unfortunately, low training MSE does not guarantee low test MSE, as shown in the graph below.

Untitled