~Ammar

LoRA is an efficient fine tuning technique that is widely used to fine tune LLMs and other Large Models for specific tasks.

The Problem:

We have an LLM which is pre-trained on a large corpus. We want to fine tune it for our specific task. These tasks can be Legal report generation, Financial summaries or Medical record analysis. However, LLMs are huge and Fine Tuning them can be computationally or financially unfeasible. Many different methods are thus devised to fine tune them efficiently. Two popular ones are Adapter Layers — addition of trainable layers in the network and Prefix Tuning — Optimize the prompt itself. However, these two have drawbacks of their own, adapter layers increase the computational cost and latency of the model while prompt optimization is difficult to achieve and causes inefficiency.

Learned over-parametrized models reside on a low intrinsic dimension:

If we have model that is parameterized by ‘P’ no. of parameters in P dimensional space. Then given that P is large that is the model is over-parameterized, the specific sets of parameters that the training process converges to (i.e., the learned models) do not occupy this entire vast space. Instead, these successful learned models tend to lie on or near a much lower-dimensional structure or manifold within that high-dimensional parameter space.

You can think of a piece of paper hanging in a large room. The room is the parameterized space while the paper is the learned model.

LoRA leverages these low lying structures to specifically update these parts of the network instead of changing all the parameters. If the entire model is parameterized by $W^{d \times k}$ then the Low rank adaptation of its update $\Delta W^{d\times k}$ can be factored into $A^{d\times r} \times B^{r\times k}$ where $r<< \min(d,k)$. This reduces the total no. of parameters to update. These matrices are inserted throughout the network to create a parallel path for the gradients to flow.

Lora.jpeg

Hence our output becomes:

$h= W_0x+ ∆Wx= W_0x+ BAx$

Toy Example:

Imagine a single weight matrix $W_0$ in a neural network layer with dimensions, say, 1000×500.

If we were to fine-tune this layer directly, we would be updating all 500,000 parameters.

Now, let's apply LoRA. We hypothesize that the update ΔW needed for a new task can be approximated with a very low rank, say r=4.