Statistical Divergences: Concepts, Applications, and Future Perspectives

Introduction

Statistical divergences are functions that quantify how one probability distribution differs from another, or more generally how one point or dataset differs from another based on some underlying measure. Unlike true metrics, divergences need not be symmetric and often lack the triangle inequality, but they provide useful “distance-like” measures of dissimilarity . In machine learning and statistics, divergences play a central role in comparing models to data, defining loss functions, and guiding optimization algorithms. Different divergences capture different aspects of discrepancy – some emphasize geometric differences, others focus on information content or support overlap. This makes each divergence suited to particular real-world use cases and sensitive to different properties of high-dimensional data.

This essay surveys several major divergences: Bregman divergences, f-divergences, the Kullback–Leibler (KL) divergence, the Jensen–Shannon (JS) divergence, the Sinkhorn divergence, and the Maximum Mean Discrepancy (MMD). For each, we outline the mathematical definition and core properties, highlight common applications in machine learning and optimization, and venture into less-explored or speculative applications – for example, in topological data analysis, non-Euclidean latent manifolds, or synthetic data with unusual distributions. We also discuss how each divergence relates to compressibility, interpretability, and generalization in high-dimensional settings. By understanding the strengths and nuances of these divergences, we can better match them to both current and future problems, including scenarios where data distributions may behave in very “strange” ways (from alien manifolds to AI alignment distributions).

Bregman Divergences

Definition and Core Properties: A Bregman divergence is defined in terms of a convex function and generalizes many familiar distance measures. Formally, given a continuously differentiable, strictly convex function $F$ on a convex set $\Omega$, the Bregman divergence between points (or distributions) $p, q \in \Omega$ is defined as:

D_F(p, q) = F(p)\;-\;F(q)\;-\;\langle \nabla F(q),\,p - q\rangle,

the difference between $F(p)$ and the first-order Taylor expansion of $F$ around $q$ evaluated at $p$ . This construction yields a nonnegative divergence $D_F(p,q)\ge 0$, which equals zero if and only if $p=q$ (assuming strict convexity of $F$) . Bregman divergences include a wide range of examples; for instance, if $F(x)=|x|^2$ (the squared Euclidean norm), then $D_F(p,q)=|p-q|^2$ is the squared Euclidean distance . If $F(p)=\sum_i p(i)\log p(i)$ (the negative of Shannon entropy), the induced Bregman divergence is (up to constant terms) the Kullback–Leibler divergence . Key properties of Bregman divergences include:

• Asymmetry: In general $D_F(p,q) \neq D_F(q,p)$; swapping arguments corresponds to using a different convex function (the Legendre dual of $F$) and typically gives a different value .
• No triangle inequality: Bregman divergences are not metrics and do not satisfy the triangle inequality . However, they do satisfy a generalized Pythagorean theorem in the geometry induced by $F$ .
• Convexity: $D_F(p,q)$ is convex in its first argument $p$ (and typically not convex in $q$) . This makes many optimizations involving a fixed $q$ tractable.
• Mean as minimizer: Uniquely for Bregman divergences, the arithmetic mean minimizes expected divergence. In fact, if a “distance” on a space has the property that for any distribution of points the mean is the minimizer of average distance, that distance must be a Bregman divergence . This remarkable property (proved by Banerjee et al. 2005) underpins the use of Bregman divergences in clustering and centroid-based methods.

Use Cases in ML and Optimization: Bregman divergences appear in many algorithms as a natural distance for comparing data points or parameters:

• Clustering: The k-means algorithm can be generalized by using a Bregman divergence instead of Euclidean distance. In Bregman clustering, each cluster’s “centroid” is the minimizer of average Bregman divergence (which, by the mean-as-minimizer property, is typically the mean under an appropriate parameterization) . For example, using the KL-divergence yields clusters optimized for “distance” in relative entropy terms, suitable for data in the form of probabilities.
• Mirror descent and optimization: Bregman divergences form the backbone of mirror descent and related algorithms . In mirror descent, one performs gradient updates in the dual space defined by a convex function $F$, and distances in the primal space are measured by $D_F$. This allows using non-Euclidean distances (like KL) to constrain updates, which can improve convergence on probability simplices or other constrained domains . Many machine learning optimization routines (e.g. the hedge algorithm in online learning) can be seen as mirror descent with a specific Bregman divergence.
• Exponential family and geometry: Every regular exponential family distribution has a unique Bregman divergence associated with its log-likelihood geometry . In fact, the log-likelihood of an exponential family can be written as $-;D_F(p,q) + \text{constant}$, meaning the best-fit parameters minimize a Bregman divergence to the empirical distribution . This connection is exploited in algorithms like principal component analysis for exponential families (EPCA), where one uses Bregman divergences to generalize PCA beyond Gaussian data . Information geometry interprets Bregman divergences as generating a dually flat manifold structure, allowing generalized projections and Pythagorean relations analogous to least-squares .

Speculative and Less Common Applications: Because Bregman divergences do not require a true metric structure, they can be applied in novel settings:

• Topological Data Analysis (TDA): Recent research has extended TDA to work with Bregman divergences instead of standard distances . This is significant because classical TDA (persistent homology, etc.) assumed metric distances, limiting analyses to Euclidean or other metric spaces. Using Bregman divergences enables computing shapes and holes in data when similarity is measured, for example, by KL-divergence or other non-metric criteria . Edelsbrunner and Wagner (2017) show that one can construct Čech and Delaunay complexes using Bregman balls, enabling persistent homology on high-dimensional data with non-Euclidean dissimilarities . This opens the door to analyzing datasets in terms of information-based geometry (like relative entropy) and could be pivotal for future data domains where an information geometry distance is more natural than Euclidean distance (for instance, comparing probability distributions attached to data points).
• Non-Euclidean Latent Spaces: In modern representation learning, latent vectors might live on curved manifolds or have intrinsically non-Euclidean structure. A carefully chosen Bregman divergence can act as a problem-specific distance in such spaces. For example, if a latent space is hyperbolic, one might use a convex function that induces a divergence aligning with the hyperbolic geometry. While this is an open research area, one can imagine using a learned convex function $F$ so that $D_F(p,q)$ captures meaningful differences on a strange manifold, something traditional distances cannot easily do. There is even work on Bregman–Wasserstein divergences that fuse optimal transport with Bregman structure to compare datasets in a geometrically informed way , hinting at future applications where data have both geometric and probabilistic complexities.