ZIXUAN WANG*, 2026

<aside> 📌

This is a personal technical writeup. Work done during an internship on MiroMind RL team. Technical writeup only for personal learning

</aside>

<aside> 🧊

TL;DR In every modern RL framework, one engine samples your rollouts (vLLM / SGLang) and a different engine learns from them (FSDP / Megatron). They share the same weights $\theta$ but not the same arithmetic, so they are two different policies, and training silently goes off-policy.

• In dense models this gap is small and continuous.
• In Mixture-of-Experts (MoE) models the router makes the gap discrete: the two engines literally route the same token to different experts, and the heavy tail of importance ratios explodes.
• In multi-turn agentic (ReAct) rollouts the error compounds over thousands of tokens and dozens of tool turns, so a source-level fix matters more as the horizon grows.
• Rollout Routing Replay (R³) record which experts the inference engine fired per token and replay that exact routing in the training forward pass, killing the mismatch at its source. </aside>

Color convention

Why this matters

Here is the entire problem in one inequality. Two engines share parameters $\theta$, but differ in kernels, precision, and MoE discrete routing, so they realize two distinct distributions:

$$ \underbrace{\textcolor{red}{\pi_{\mathrm{infer}}}(\cdot\,;\theta)}{\text{sampler rolls out (vLLM / SGLang)}} \;\neq\; \underbrace{\textcolor{blue}{\pi{\mathrm{train}}}(\cdot\,;\theta)}_{\text{learner scores \& differentiates (FSDP / Megatron)}} $$

even though both are "the model with weights $\theta$." A numerically tiny per-token disagreement is, formally, an off-policy bug. The roadmap:

1. Rollout Routing Replay (R³) training for MoE models: fix the mismatch at its source.
2. Interaction Scaling in ReAct agents: why long horizons make the source-level fix indispensable.

<aside> 🎨

• the engine ratio $w_t=\textcolor{blue}{\pi_{\mathrm{train}}}/\textcolor{red}{\pi_{\mathrm{infer}}}$
• the temporal ratio $\rho_t=\textcolor{blue}{\pi_{\mathrm{train}}}(\theta)/\textcolor{blue}{\pi_{\mathrm{train}}}(\theta_{\mathrm{old}})$ </aside>

Figure 1. Rollout Routing Replay (R³) at a glance. The $\textcolor{red}{\mathsf{Record}}$ (inference) → $\textcolor{green}{\mathsf{Replay}}$ (bridge) → $\textcolor{blue}{\mathsf{Train}}$ flow: the inference engine records its per-token top-$K$ expert mask during rollout, that exact mask is replayed into the training forward pass, so the learner scores precisely the experts that fired. (Mechanism detailed in Part 3.)

1. The Mismatch Problem

1.1 The ideal: on-policy REINFORCE

Vanilla policy gradient for a response $a$ with reward $R(a)$:

$$ \theta \leftarrow \theta + \mu\, \underbrace{\mathbb{E}{a\sim \textcolor{blue}{\pi{\mathrm{train}}}(\theta)}\!\Big[\,R(a)\,\nabla_\theta \log \textcolor{blue}{\pi_{\mathrm{train}}}(a;\theta)\,\Big]}_{\text{on-policy: sample and score with the SAME distribution}} $$

This estimator is unbiased only because the sampling distribution and the scored distribution coincide.

1.2 The bug: hybrid engines make it off-policy