The most importat concepts in selecting a statistical learning procedure for a specific data set.
Evaluating the performance of a statistical learning method on a given data set
= Measurement of how well predictions match observations
MSE is the most commonly used measure for regression.
$$ MSE=\frac1n\underset{i=1}{\overset{n}\Sigma}(y_i-\hat f(x_i))^2 $$
where
$Training MSE$
→ computed using the training data, data used to fit the model
→ $\hat f(x_i) \sim y_i$
$TestMSE$
→ computed using test data, data previously unseen
→ $Ave(y_0-\hat f(x_0))^2$
→ $(x_0, y_0)$: unseen and unused data
We are more interested in test MSE in measuring the quality of the method. However, if no test observations are available(no test data), then one might select a method that minimizes the train MSE.
Unfortunately, low training MSE does not guarantee low test MSE, as shown in the graph below.
