https://arxiv.org/abs/2410.12341 (surplexity)

KEVIN FRANS latest post

Short version: “maximize surprisal” only makes sense once you say which surprisal. On the Fisher-information manifold, three different beasts show up:

1. Pointwise surprisal

   s(x;\theta) = -\log p_\theta(x).

   Maximizing this with respect to \theta for a fixed datum is just “make the datum as unlikely as possible.” That’s anti-learning. Geometrically, it means walking up the negative log-likelihood, i.e., moving away from the data along the manifold. Cute for trolling your own model, useless for inference.

2. Expected surprisal (entropy)

   H(\theta)=\mathbb{E}{x\sim p\theta}[ -\log p_\theta(x)].

   Maximizing H is the max-entropy principle. On the statistical manifold, the Fisher metric I(\theta) sets the local curvature of KL, and negative entropy is a convex potential in exponential families. So “maximize surprisal” here means: move along the manifold toward higher-entropy members of your family (subject to constraints). The Fisher metric doesn’t change the target, it sets the geometry of the path (natural gradient steps: \tilde{\nabla} H = I(\theta)^{-1}\nabla H).

3. Bayesian surprise / information gain

   \mathrm{IG} = \mathbb{E}{x}\big[ D{\mathrm{KL}}(p(\theta\!\mid\!x)\,\|\,p(\theta))\big].

   This is the exploration one actually wants. In the small-step (Laplace) regime,

   \mathrm{IG} \;\approx\; \tfrac12 \log\det\!\big(I(\theta)\,\Sigma_{\text{prior}}\big) \quad \text{or} \quad \tfrac12 \mathrm{tr}\!\big(I(\theta)\,\Sigma_{\text{prior}}\big),

   depending on the approximation. Translation: maximizing expected “surprise” of your beliefs routes you to regions with large Fisher curvature. That’s exactly D-optimal/A-optimal design: choose experiments that maximize \det I or \mathrm{tr}\,I. On the manifold, you’re steering toward the sharpest curvature directions because they promise the biggest posterior update.

How this fits together

• Fisher information: I(\theta)=\mathbb{E}[\,\nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top\,]. It’s the local quadratic form of KL and the Riemannian metric of the model manifold.
• Minimizing expected surprisal of data (i.e., doing MLE) = gradient flow on this manifold if you use natural gradients.
• Maximizing Bayesian surprise = target data that increase curvature-weighted learning the most, i.e., where I(\theta) is big in prior-uncertain directions.

If you’re thinking in predictive-processing/Friston terms: minimizing sensory surprisal is the exploitation half (natural-gradient descent on free energy); maximizing expected Bayesian surprise is the exploration half (information-seeking), and its local surrogate is exactly Fisher. Maximize the wrong thing (pointwise data surprisal) and you just train your smoke alarm to cheer for fires.

Short version: surprisal = −log p(x|θ). The Fisher information matrix (FIM) is the expected curvature of surprisal with respect to θ. So Fisher tells you how quickly surprisal rises when you nudge parameters. Big Fisher eigenvalues mean tiny parameter changes cause big jumps in surprisal; small ones mean the model is basically shrugging.

Links to surprisal

• Gradient: ∇θ surprisal = − score.
• Curvature: F(θ) = E[∇θ surprisal · ∇θ surprisalᵀ] = E[∇²θ surprisal].