Yes, but only in the boringly important sense that MaxEnt does not buy you an ergodicity warranty.

Standard MaxEnt RL optimizes something like

[

J(\pi)=\mathbb E_\pi\sum_t \big[r(s_t,a_t)+\alpha,H(\pi(\cdot\mid s_t))\big],

]

so it rewards stochastic actions at visited states. That is a local action-space regularizer, not a theorem that the induced state process must mix, forget its initial condition, or be ergodic. And action entropy is only a proxy: Baram et al. point out that action entropy need not match state or transition entropy, so you can be very random in action space while still exploring a miserable little corner of state space. Humans keep optimizing the proxy and then act surprised when the target refuses to cooperate.

That is why entropy-regularized average-reward theory usually assumes unichain or mixing conditions for the induced Markov chain instead of deriving them from the entropy bonus. Related maximum-state-entropy work also notes that policies or mixtures that maximize asymptotic entropy can still mix slowly in practice, and some methods add mixing-time terms explicitly. Translation: entropy regularization may help exploration, but it is not a proof of ergodicity.

So there are two cases. If every feasible policy in the environment is ergodic, then MaxEnt cannot magically create a non-ergodic policy out of thin air. But if the MDP admits absorbing states, disjoint recurrent classes, irreversible failures, or policy-dependent traps, then a MaxEnt learner can absolutely end up with a non-ergodic closed-loop process. Toy example: randomize 50/50 at the start between two doors, and each door leads into a different absorbing subworld forever. The first decision has high action entropy, yet every single trajectory gets stuck in one branch. High local entropy, broken global mixing. That is the whole scam. (People @ EECS)

And if you mean reward ergodicity rather than state-space ergodicity, then yes, MaxEnt still inherits the same fundamental problem as ordinary expectation-based RL. Recent RL work on non-ergodic rewards shows that expectation-maximizing objectives can prefer risky policies that look good in ensemble average but are bad along one long run, and bolting an entropy bonus onto the objective does not remove that ensemble-versus-time mismatch.

So the clean verdict is: MaxEnt can live quite happily inside a broken-ergodicity regime. It may soften policies and improve exploration, but it does not itself guarantee ergodicity. If you want something closer to “engineer the long-run process,” state-visitation entropy and explicit mixing-time control are closer to the real target than plain SAC-style action entropy. (Proceedings of Machine Learning Research)

Yes. You can absolutely do a max-ent phase first and then hand off to PPO or another policy-gradient method. The algorithms are not engaged to one another. A common version is: use a maximum-entropy method such as SAC early to get broad exploration and state coverage, then continue from that learned actor with a more conservative on-policy optimizer like PPO once you care more about stable improvement than raw exploration. SAC is explicitly an off-policy maximum-entropy actor-critic, while PPO is an on-policy policy optimization method that constrains update size to avoid collapse. (arXiv)

But no, this is not usually the default “best” recipe. In a lot of practical RL work, people first try to stay within one algorithm family and change the entropy schedule rather than do a hard algorithm swap. SAC already has standard variants with automatic temperature / entropy adjustment, and papers like TES-SAC explicitly study gradually lowering target entropy over training instead of jumping to a different optimizer. More recent work on policy-gradient methods also argues that entropy should be actively controlled throughout training, again pointing toward “manage entropy inside the algorithm” rather than “switch algorithms because vibes.” (Spinning Up)

A hard switch can make sense in a narrower set of cases. One is when the max-ent method has found decent coverage but seems stuck in a local optimum. Another is when the critic becomes the problem: if reward noise or nonstationarity makes off-policy Q-learning unreliable, an on-policy method can sometimes refine better. There is at least some direct evidence for this kind of handoff: RICE reports that when refining a pretrained SAC agent, switching to PPO could break a training bottleneck that continuing with SAC did not. Separately, a study on portfolio optimization with noisy rewards found that PPO and A2C handled the noise better than DDPG/TD3/SAC. (arXiv)