Date: January 18, 2024

Topic: Optimizers

Recall

Optimizers try to find the minimum value of a function through gradient descent

Notes

Optimizers

• Find minimum values of functions
• Build parameterized models based on data
• Refine allocations to stocks in portfolios

Using optimizers

1. Provide a function to minimize (e.g., $f(x) = x^2 +0.5$)
2. Provide an initial guess
3. Call the optimizer

Gradient descent

• Using a random value, find the gradient and see where the optimizer should head to find the min.

Minimizers can be used with the scipy.optimize library

Minimizing with Python

• Can be done with scipy.optimize library

import scipy.optimize as SPO

def f(x):
	Y = (X-1.5)**2 + 0.5
  return Y

# First, we pass in a guess
x_guess = 2.0

# Then, call the optimizer and pass in the function defined by f(x)
min_result = spo.minimize(f, x_guess, method='SLSQP', options={'disp': True})

# Print the minima coords
print(min_result.x) # x_coord
print(min_result.fun) # y_coord

# Plot function values, mark minima
Xplot = np.linspace(0.5, 2.5, 21)
Yplot = f(Xplot)
plt.plot(Xplot, Yplot)
plt.plot(min_result.x, min_result.fun, 'ro')
plt.title("Minima of an objective function")
plt.show()

Given a line drawn by any 2 points on a graph, if the line is above the graph then it is a convex problem

Convex Problems

• Choose any 2 points and draw a line
• If the line is above a graph, then it is a convex problem (1 local minima which is also global)

Optimizers need a problem to minimize, hence for linear regression, we can provide it with the sum of squared residuals.

Building a parametrized model

• We need to know what we are minimizing (for e.g., finding the minima in the above examples)

Minimizing Error

• For a line $f(x) = C_0x + C_1$ (y=mx+c)
• We want to find the best fit line according to the data points
• Hence, we need to minimize the errors ($e_0, e_1, ...$)
  • Possible formulas: $\sum_i\lvert e_i\rvert$ (sum of absolutes) , $\sum_i{e_i^2}$ (sum of squared residuals)

<aside> 📌 SUMMARY: Minimization can be done in many dimensions, and one popular thing to minimize is the sum of squared errors to find a good line fit

</aside>