Date: May 19, 2024

Topic: Introduction to Regression

Recall

In regression, we want to find the best function that approximates our data points.

This is done by choosing a function that reduces the amount of squared errors between the approximation and the data points

Notes

Linear Regression

Regression

• Mapping continuous inputs to continuous outputs (instead of discrete outputs)
• When we regress, we tend to fall back towards the mean

Linear Regression

We find a linear function that best fits actual data, minimizing the sum of squared errors

The green line represents the best fit line minimizing the data and function squared errors

The best constant function can be found through using differentiation, where the rate of change at minimum error would be 0

Varying $k$ lets us use higher order functions that can fit the data better

However, if the order is too high the predictions become unrealistic

For this dataset, we see that the best degree is cubic as the predictions remain realistic while minimizing the errors.

Finding the best constant function ($c$)

Given a 2D relationship between $x$ and $y$, we use the sum of squared residuals as it can be differentiated

• $f(x) = c$
• $E(c)=\sum^{n}_{i=1}(yi-c)^2$
• $\frac{dE(c)}{dc}=\sum^n_{i=1}2(yi-c)(-1)$

Hence, to minimize the error term, the change of error = 0

• $-\sum^n_{i=1}2(yi-c)=0$
• $\sum^n_{i=1}yi-\sum^n_{i=1}c = 0$
• $n\cdot c=\sum^n_{i=1}yi$

Thus, we get the best constant as $c=\sum^n_{i=1}\frac{yi}{n}$ (or the mean of the $y$ values)

• ie, the best guess for $y$ without any $x$ is the mean of all $yi$

Order of Polynomial

The general form of a function: $f{x} = c_0+c_1x+c_2x^2+...+c_kx^k$

• $k=0$: Constant
• $k=1$: Line
• $k=2$: Parabola

Comparing Graphs

• The above graph shows three different orders for $k=0$ to $k=2$

• Going even further, we can increase until $k=9$, where every datapoint is hit
• However this provides very bad estimations at certain unknown sizes like $9000$
• In this scenario, we have overfit the training data

Comparing degrees with data

<aside> 📌 SUMMARY: We want to find the right degree which gives us the lowest error for our model against the data

</aside>