Scalar Valued Function of Single Variable

Function takes in a scalar and maps to a scalar

Scalar Valued Function of Multiple Variables

Function takes in a vector and maps to a scalar (scalar-by-vector $\frac{\partial s}{\partial v}$)

Computationally, the denominator and numerator forms are equivalent.
The notation we use is just important for aligning shapes when doing matmul.

Vector Valued Function of a Single Variable

Function takes in a scalar and maps to a vector (vector-by-scalar $\frac{\partial v}{\partial s}$)

To convert between numerator and denominator layout, we just need to transpose the former to get the latter.

Vector Valued Function of Multiple Variables

Function takes in a vector and maps to another vector (vector-by-vector)

Numerator Layout

Num. rows equals to num. of numerators (or num. of functions here)

2 numerators — f1 and f2, so 2 rows

Denominator Layout

Num. rows equals to num. of denominators (or num. of variables)

3 denominators — x1, x2, x3, so 3 rows

Scalar Valued Function of Many (Matrix) Variables

Function takes in a matrix and maps to a scalar (scalar-by-matrix $\frac{\partial s}{\partial M}$)

Matrix Valued Function of a Single Variable

Function takes in a scalar and maps to a matrix (matrix-by-scalar)

We can use vectorization on the input $x$ so that we get a Jacobian form that is easier to work with.

<aside> 📌 SUMMARY: Matrix calculus of different forms. Take note of the use of flattening matrices to vectors for easier computation, and the uniqueness of element-wise functions where the Jacobian result is a diagonal matrix

</aside>