Information geometry

In mathematics and especially in statistical inference, information geometry is the study of probability and information by way of differential geometry. It reached maturity through the work of Shun'ichi Amari in the 1980s, with what is currently the canonical reference book: "Differential-geometrical methods in statistics".

Introduction

The main tenet of information geometry is that many important structures in probability theory, information theory and statistics can be treated as structures in differential geometry by regarding a space of probabilities as a differential manifold endowed with a Riemannian metric and a family of affine connections. For example,
* The Fisher information metric is a Riemannian metric.
* The Kullback-Leibler divergence is one of a family of divergences related to dual affine connections.
* An exponential family is flat submanifold under the e-affine connection.
* The maximum likelihood estimate is a projection under the m-affine connection.
* The unique existence of maximum likelihood estimate on exponential families is the consequence of the e- and m- connections being dual affine.
* The EM algorithm is, under broad conditions, an iterative dual projection method under the e-connection and m-connection.
* The concepts of accuracy of estimators, in particular the first and third order efficiency of estimators, can be represented in terms of imbedding curvatures of the manifold representing the statistical model and the manifold of representing the estimator (the second order always equals zero after bias correction).
*The higher order asymptotic power of statistical test can be represented using geometric quantities.

The importance of studying statistical structures as geometrical structures lies in the fact that geometric structures are invariant under coordinate transforms. For example, a family of probability distributions, such as Gaussian distributions, may be transformed into another family of distributions, such as log-normal distributions, by a change of variables. However, the fact of it being an exponential family is not changed, since the latter is a geometric property. The distance between two distributions in this family defined through Fisher metric will also be preserved.

The statistician Fisher recognized in the 1920s that there is an intrinsic measure of amount of information for statistical estimators. The Fisher information matrix was shown by Cramer and Rao to be a Riemannian metric on the space of probabilities, and became known as Fisher information metric.

The mathematician Cencov (Chentsov) proved in the 1960s and 1970s that on the space of probability distributions on a sample space containing at least three points,
* There exists a unique intrinsic metric. It is the Fisher information metric.
* There exists a unique one parameter family of affine connections. It is the family of $alpha$-affine connections later popularized by Amari.Both of these uniqueness are, of course, up to the multiplication by a constant.

Amari and Nagaoka's study in the 1980s brought all these results together, with the introduction of the concept of dual-affine connections, and the interplay among metric, affine connection and divergence(information geometry). In particular,
* Given a Riemannian metric "g" and a family of dual affine connections $Gamma\_alpha$, there exists a unique set of dual divergences $D\_alpha$ defined by them.
* Given the family of dual divergences $D\_alpha$, the metric and affine connections can be uniquely determined by second order and third order differentiations.Also, Amari and Kumon showed that asymptotic efficiency of estimates and testscan be represented by geometrical quantities.

Basic concepts

* Statistical manifold: space of probability distribution, statistical model.
* Point on the manifold: probability distribution.
* Coordinates: parameters in the statistical model.
* Tangent vector: Fisher score function.
* Riemannian metric: Fisher information metric.
* Affine connections.
* Curvatures: associated with information loss
* Information divergence.

Fisher information metric as a Riemannian metric

Information geometry is based primarily on the Fisher information metric:

:$g\_\{ij\}=int\; frac\{partial\; log\; p(x,\; heta)\}\{partial\; heta\_i\}\; frac\{partial\; log\; p(x,\; heta)\}\{partial\; heta\_j\}\; p(x,\; heta),\; dx.$

Substituting "i" = −log("p") from information theory, the formula becomes:

:$g\_\{ij\}=int\; frac\{partial\; i(x,\; heta)\}\{partial\; heta\_i\}\; frac\{partial\; i(x,\; heta)\}\{partial\; heta\_j\}\; p(x,\; heta),\; dx.$

Intuitively, this says the distance between two points on a statistical differential manifold is the amount of information between them, i.e. the informational difference between them.

Thus, if a point in information space represents the state of a system, then the trajectory of that point will, on average, be a random walk through information space, i.e. will diffuse according to Brownian motion.

With this in mind, the information space can be thought of as a fitness landscape, a trajectory through this space being an "evolution". The Brownian motion of evolution trajectories thus represents the "no free lunch" phenomenon discussed by Stuart Kauffman.

History

The history of information geometry is associated with the discoveries of at least the following people, and many others
* Sir Ronald Aylmer Fisher
* Harald Cramér
* Calyampudi Radhakrishna Rao
* Solomon Kullback
* Richard Leibler
* Claude Shannon
* Imre Csiszár
* Cencov
* Bradley Efron
* Vos
* Shun-ichi Amari
* Hiroshi Nagaoka
* Kass
* Shinto Eguchi
* Ole Barndorff-Nielsen

Some applications

Natural gradient

An important concept in information geometry is the natural gradient. The concept and theory of the natural gradient suggests an adjustment to the energy function of a learning rule. This adjustment takes into account the curvature of the (prior) statistical differential manifold, by way of the Fisher information metric.

This concept has many important applications in blind signal separation, neural networks, artificial intelligence, and other engineering problems that deal with information. Experimental results have shown that application of the concept leads to substantial performance gains.

References

* Shun'ichi Amari - "Differential-geometrical methods in statistics", Lecture notes in statistics, Springer-Verlag, Berlin, 1985
* Shun'ichi Amari, Hiroshi Nagaoka - "Methods of information geometry", Transactions of mathematical monographs; v. 191, American Mathematical Society, 2000
* M. Murray and J. Rice - "Differential geometry and statistics", Monographs on Statistics and Applied Probability 48, Chapman and Hall, 1993.
* R. E. Kass and P. W. Vos - "Geometrical Foundations of Asymptotic Inference", Series in Probability and Statistics, Wiley, 1997.
* N. N. Cencov - "Statistical Decisions Rules and Optimal Inference", Translations of Mathematical Monographs; v. 53, American Mathematical Society, 1982