Authors: Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S. Dhillon, and Devvrit Khatri.

<aside> 💡

tl;dr

As Continual Learning becomes central to training Large Language Models (LLMs), several question remains unclear:

• How do we inject new knowledge and skills into these model without forgetting the prior ones; and how do we do this at scale?
• Can offline data or off-policy trajectories improve performance without a model collapse?
• Can the geometry of online methods (RL, OPD) which are not prone to forgetting inspire new approaches?

We introduce subspace tuning (SST) a practical recipe for learning from off-policy data reliably. Our method can be used on top of any finetuning method (like SFT or self-distilltion (SDFT)) to instantly improve its learning-forgetting pareto. We show that SST scales gracefully as the number of parameter in the model and the amount of offline data increases.

This work provides a new direction to look at the update geometry and not just the loss or data.

</aside>

Why Forgetting hinders Continual Learning

A pretrained model’s weights support many behaviors at once: reasoning, coding, instruction following, and more. Offline data provides rich knowledege and task abilities, however performing simple SFT on offline data causes severe degredation in general model performance [1]. As agentic infrastructure scales, learning from offline trajectories becomes harder due to the off-policy nature of these traces causing severe collapse under token-level supervised training. To enable true continual learning, mitigating forgetting remains a key hurdle.

On-Policy Learning geometry inhibits Forgetting

On-policy RL also stays closer to the base policy and forgets less than SFT [2]. Recent works such as [3] shows the Spectral geometry under SFT vs. RLVR on Qwen3-8B:

RLVR maintains a stable top-k spectrum with minimal subspace rotation, unlike SFT which shows massive deviations. We posit that this contrast in the update geometry is the key to mitigate forgetting in SFT. Intuitively, updating the top-k directions of any weight matrix will cause severe divergence in it’s behavior from the base model, which in turn can cause catastrophic forgetting. Can we exploit this update geometry to learn from offline data…

Subspace Tuning

We intend to mimic the update geometry of RLVR in supervised training. For a weight matrix W, let $U_k$ and $V_k$ contain its top-k left and right singular vectors. SST projects the gradient update outside the top-k spectrum:

$$ \Pi(\Delta W) = (I - U_k U_k^\top)\,\Delta W\,(I - V_k V_k^\top) $$

and doing so, pushes existing offline-data training recipes to a better learning-forgetting pareto:

Results on Qwen3-8B, averaged over four domains. Both axes show changes from the base model in percentage points (pp); higher is better. Circles denote training without SST, stars denote training with SST, and the black triangle marks the base model.

But restricting updates to the same subspace can inhibit learning. The hyperparameter k provides a knob to control the learning-fogetting tradeoff. However, this makes the algorithm learn in a static subspace which may limit learning. As the weights move due to the the new subspace updates, we want to realign the subspace to prevent forgetting of the current model behavior. Hence, we introduce a refresh mechanism to re-orient the protected spectrum with the current weights every few (~100) steps.

After refresh, optimization subspaces is better aligned to the current weights allowing larger part of the update to survive. More frequent refresh opens new spectral directions but mildly weakens protection relative to the original model. Repeated refreshes can stitch together accepted steps from changing, locally constrained subspaces instead of forcing the whole run through one subspace chosen at the start. The overall algorithm with the refresh mechanism is shown below -

Uₖ, Vₖ = top_k_svd(W)
for step in training_steps:
    ΔW = optimizer.propose(g)
    ΔW_free = (I − UₖUₖᵀ) ΔW (I − VₖVₖᵀ)
    W += ΔW_free
    if step is a refresh:
        Uₖ, Vₖ = top_k_svd(W)

Scaling

To enable true continual learning, any learning algorithm should generalize as the amount of training data and the number of parameters scale. We observe that SST strengthens with model scale. In five-epoch fine-tuning, unconstrained SFT loses roughly 30–40 points of prior capability from 1.7B through 14B. SST approaches each model’s base capability as scale grows while reaching higher target accuracy. We make similar observtions as we scale the amount of training data too!

Scaling training data. Qwen3-8B fine-tuned on Science under a matched 45k-example budget: the novel arms see a fresh stream once, the repeated arms cycle 9k examples for five epochs. Under both exposures SFT drifts progressively farther from the base model, while SST stays flat and learns the task better.

Scaling model size. Qwen3 models from 1.7B to 14B fine-tuned on Science for five epochs. SFT forgets heavily at every scale, whereas forgetting under SST shrinks as models grow and nearly vanishes from 8B onward. SST also reaches higher new-task accuracy at every size.