nowhere else to frictionlessly c/p them so here!!

Short answer: the detour started in the 1920s–30s when Fisher popularized the p-value and 0.05 “felt right,” then Neyman–Pearson turned testing into a ritual with Type I/II errors. After World War II, journals, textbooks, and regulators fossilized it. By the time anyone asked “does this generalize out of distribution?” the field was already worshipping a threshold. The backlash only went mainstream in the 2010s with the ASA statement and the “retire statistical significance” push.

Here’s a clean timeline so you can sigh efficiently:

• 1900–1933: Pearson χ² and Fisher formalize tail-area p-values; Neyman–Pearson codify decision rules. Bright-line thinking is born.
• 1950s–70s: “Null-hypothesis significance testing” becomes default; Tukey and Huber launch robust stats as the first “we don’t trust your assumptions” movement.
• 1990s: Multiple testing explodes; FDR control shows up so people stop declaring victory 10,000 times per paper.
• 2010s: Reproducibility crisis; ASA’s 2016 statement and Nature’s 2019 editorial tell everyone to stop treating p < 0.05 like a magic spell. Culture change begins, slowly, because inertia loves tenure.

Now to the part you actually care about: generalization and distribution shift. If you want statistics to do adult work instead of cosplaying as a gatekeeper, use tools built for shift, not just for in-distribution averages.

What to do instead of genuflecting to p

1. Evaluate on prediction, not thresholds.

   Use proper scoring rules (log loss, Brier), cross-validation, and out-of-sample curves. Then show uncertainty with intervals rather than a binary “significant/not.” The ASA explicitly recommends moving past dichotomies.

2. Detect distribution shift explicitly.

• Two-sample tests: kernel MMD to compare train vs. deploy distributions. Cheap, multivariate, and assumption-light.
• Monitoring metrics: PSI as a quick dashboard; it’s crude but serviceable for ongoing drift alerts in production. Pair with KL or JS if you’re feeling fancy.
• Change-point detection: CUSUM/Page tests for “something just snapped” moments in time series.

1. Weight the data by where you’re going, not where you’ve been.

   Under covariate shift, reweight each example by a density ratio w(x)=p_{\text{test}}(x)/p_{\text{train}}(x) and minimize importance-weighted risk. Use uLSIF/IWCV to estimate the ratio and keep weights clipped so a few aliens don’t hijack the loss. Translation: anomalies that look like the new world get upweighted; random junk doesn’t.

2. Be robust when anomalies might be signal, not noise.

   Swap the loss: Huber/Tukey M-estimators, quantile loss, trimmed/Winsorized means. These reduce leverage of one-off nonsense while still letting consistent edge cases move the model. The 1960s already gave you this; you’re allowed to use it.