RL notes | Notion

Short answer: yes—quality-diversity (QD) methods deliberately maximize behavioral coverage, so they run “hotter” entropy than most RL that’s optimized to exploit a single peak. If you want generalization or grokking, you need a wide search, not a monoculture.

Here’s a practical ranking by effective search entropy (think: entropy of behaviors visited and retained, not just the policy’s action noise). Highest at the top:

Tier	Methods	Why the entropy is high/low
S (scorching)	POET, MAP-Elites, Novelty Search (+ Local Competition), QD-RL, Go-Explore (phase 1/archive)	Explicitly optimize for diversity across a behavior space and keep an archive. Maintains many niches simultaneously; maximal state-visitation and repertoire entropy.
A (very high)	Unsupervised/skill-diversity RL: DIAYN, APS/CIC/APT, empowerment/MI objectives; curiosity/count-based: ICM, RND, pseudo-counts	Intrinsic objectives reward novelty or mutual information, producing broad coverage even without extrinsic reward. Often run multiple skills/options → high repertoire entropy.
A− / B+	Max-entropy control: SAC (with temperature tuning), MPO with entropy/KL constraints; uncertainty-driven exploration: Bootstrapped/Thompson DQN	SAC literally optimizes return + α·H(π); target entropy keeps policies stochastic. Bootstrapped heads approximate posterior sampling → diverse trajectories.
B (moderate)	Model-based RL w/ ensembles & exploration: PETS/MBPO (UCB/TS bonuses), Dreamer-v2/3 with disagreement; MuZero (with Dirichlet at root)	Models plus epistemic bonuses explore more than plain policy gradient, but planning/MPC can still prune action entropy; diversity depends on ensemble width and bonus strength.
C	On-policy PG with simple entropy bonus: PPO, A2C, TRPO	Maintains policy entropy early, but on-policy data is narrow and bonuses decay. Coverage OK in easy tasks, mediocre OOD.
C− / D+	Distributional RL (C51, QR-DQN) without special exploration; vanilla DQN with ε-greedy	Better value estimates, same old exploration. ε-greedy is low-quality noise, not structured coverage.
D (low)	Deterministic actor-critic: DDPG/TD3 (without exploration tricks)	Fast exploiters, brittle explorers. Action noise ≠ behavioral diversity.
F (entropy diet)	Offline RL on narrow logs, Behavior Cloning/DAgger, DPO/RLHF-style preference fitting with tight KL	You’re fitting to a thin dataset or a reference model; objectives and KL constraints collapse policy entropy and prune tails. Great for stability, terrible for coverage.

Quick notes so no one hyperventilates:

QD vs RL: QD maximizes repertoire entropy over a descriptor space, not just action entropy. That’s why it wins out-of-distribution: it keeps many partial solutions alive.
SAC caveat: if you crank α down or auto-tune to a low target entropy, SAC slides toward PPO-level entropy. The ranking assumes a sensible target entropy.
Model-based: ensembles and disagreement help, but planners happily route around uncertainty and kill entropy if your bonuses are timid.
On- vs off-policy: off-policy can accumulate a higher-entropy replay if your exploration is good; otherwise it just replays the same narrow junk faster.
DPO/RLHF: preference optimization with a tight KL is entropy-suppressing by design. You can raise temperature or loosen KL, but the default is conservative.

If your goal is OOD robustness or grokking, operate in S/A tiers during pretraining (QD, skills, curiosity), then anneal toward B/C for competence. First learn a wide library of behaviors, then specialize. Yes, it’s compute-hungry. So is failing in the wild.

Short answer: because this paper is doing on-policy RL in environments with a custom set-level objective, not offline alignment from pairwise preference data. DPO isn’t the right tool for that job.

Slightly longer, without the sugar-coating:

Problem setting mismatch. The paper trains via environment rollouts (BabyAI, Minigrid, Algorithmic Creativity), with resets and a new “polychromic” objective over sets of trajectories. That’s classic policy-gradient territory, so they adapt PPO and its clipping/KL control. DPO is designed for aligning models to pairwise human preferences from a static dataset; no environment, no advantage estimates.
Credit assignment granularity. Their method modifies the advantage function so every trajectory in a sampled set shares the same learning signal, and they optimize it with PPO. DPO gives you a sequence-level binary (or scalar) preference signal between two completions, not per-timestep advantages needed for control and exploration bonuses.
On-policy exploration mechanics. They rely on vine sampling to collect on-policy sets of rollouts and then apply PPO updates with clipping. DPO neither collects on-policy trajectories nor supports these vine/TRPO-style mechanics; it assumes you already have preference pairs.