Short answer: you mixed two levels. Per-example memorization is sharp and brittle; the aggregate weight directions that store lots of memorized junk are low-curvature.

Here’s the tidy version without hand-waving:

• Per-example view (around a single memorized sample):

  If a model has essentially memorized a point, the loss landscape near the parameters that fit that point is high curvature (sharp). Tiny parameter nudges spike the loss, which is why it’s brittle: the model clings to that datapoint like a raccoon to a trash can. Multiple papers report this sharpness signal for memorized or mislabeled points.

• Population/weight-space view (averaged over many points):

  When you average curvature across data to get a spectrum over weight directions (e.g., via K-FAC/Fisher bases), the consistent, shared mechanisms (generalization) stack coherently and live in higher-curvature directions. The idiosyncratic stuff that supports recitation points in lots of different directions, cancels in the average, and sinks into the low-curvature tail. Editing out those low-curvature components suppresses untargeted recitation while mostly keeping reasoning intact. That’s the central result of the paper you linked.

What this says about Fisher information:

• For models trained with cross-entropy, the Fisher Information Matrix (FIM) equals the expected Hessian of the negative log-likelihood, so “curvature” and “Fisher” are talking about the same beast in different outfits:

  F = \mathbb{E}[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top] = \mathbb{E}[\nabla^2_\theta(-\log p_\theta(y|x))].

  The paper explicitly uses a K-FAC approximation to analyze this spectrum.

• Per-example: memorized samples tend to have large local Fisher/curvature along the directions that overfit them. That’s the brittleness signal.

• Aggregate spectrum: the low-eigenvalue (low-Fisher) directions of the weight-space curvature basis disproportionately carry memorized, narrow features; the high-eigenvalue (high-Fisher) directions encode broadly useful structure. Prune the former, keep the latter.

So your slogan, cleaned up:

• Memorization is brittle because it’s sharp per sample, not because it’s globally high-curvature in the model’s weight basis.
• In the Fisher/curvature spectrum averaged over data, memorization mostly inhabits low-curvature, low-Fisher directions; generalization occupies higher-curvature, higher-Fisher ones. Edit there accordingly.

If only human memory came with a K-FAC slider. You’d finally forget that one horrible ninth-grade presentation and keep the useful parts like, you know, reading.