 Rényi entropy

In information theory, the Rényi entropy, a generalisation of Shannon entropy, is one of a family of functionals for quantifying the diversity, uncertainty or randomness of a system. It is named after Alfréd Rényi.
The Rényi entropy of order α, where α 0, α 1 is defined as
where p_{i} are the probabilities of {x_{1}, x_{2} ... x_{n}} and log is in base 2. If the probabilities are all the same then all the Rényi entropies of the distribution are equal, with H_{α}(X)=log n. Otherwise the entropies are weakly decreasing as a function of α.
Higher values of α, approaching infinity, give a Rényi entropy which is increasingly determined by consideration of only the highest probability events. Lower values of α, approaching zero, give a Rényi entropy which increasingly weights all possible events more equally, regardless of their probabilities. The intermediate case α=1 gives the Shannon entropy, which has special properties. When α=0, it is the maximum possible Shannon entropy, log(n).
The Rényi entropies are important in ecology and statistics as indices of diversity. The Rényi entropy also important in quantum information, it can be used as a measure of entanglement. In XY Heisenberg spin chain the Rényi entropy was calculated explicitly in terms of modular function of α.^{[clarification needed]} They also lead to a spectrum of indices of fractal dimension.^{[clarification needed]}
Contents
H_{α} for some particular values of α
Some particular cases:
which is the logarithm of the cardinality of X, sometimes called the Hartley entropy of X.
In the limit that α approaches 1, it can be shown using L'Hôpital's Rule that H_{α} converges to
which is the Shannon entropy.
Collision entropy, sometimes just called "Rényi entropy," refers to the case α = 2,
where Y is a random variable independent of X but identically distributed to X. As , the limit exists as
and this is called Minentropy, because it is the smallest value of H_{α}.
Inequalities between different values of α
The two latter cases are related by . On the other hand the Shannon entropy H_{1} can be arbitrarily high for a random variable X with fixed minentropy.
 is because .
 is because .
 since according to Jensen's inequality .
Rényi divergence
As well as the absolute Rényi entropies, Rényi also defined a spectrum of divergence measures generalising the Kullback–Leibler divergence.
The Rényi divergence of order α, where α > 0, from a distribution P to a distribution Q is defined to be:
Like the KullbackLeibler divergence, the Rényi divergences are nonnegative for α>0. This divergence is also known as the alphadivergence (αdivergence).
Some special cases:
 : minus the log probability under Q that p_{i}>0;
 : minus twice the logarithm of the Bhattacharyya coefficient;
 : the KullbackLeibler divergence;
 : the log of the expected ratio of the probabilities;
 : the log of the maximum ratio of the probabilities.
Why α = 1 is special
The value α = 1, which gives the Shannon entropy and the Kullback–Leibler divergence, is special because it is only when α=1 that one can separate out variables A and X from a joint probability distribution, and write:
for the absolute entropies, and
for the relative entropies.
The latter in particular means that if we seek a distribution p(x,a) which minimizes the divergence of some underlying prior measure m(x,a), and we acquire new information which only affects the distribution of a, then the distribution of p(xa) remains m(xa), unchanged.
The other Rényi divergences satisfy the criteria of being positive and continuous; being invariant under 1to1 coordinate transformations; and of combining additively when A and X are independent, so that if p(A,X) = p(A)p(X), then
and
The stronger properties of the α = 1 quantities, which allow the definition of conditional information and mutual information from communication theory, may be very important in other applications, or entirely unimportant, depending on those applications' requirements.
Exponential families
The Rényi entropies and divergences for an exponential family admit simple expressions (Nielsen & Nock, 2011)
and
where
 J_{F,α}(θ:θ') = αF(θ) + (1 − α)F(θ') − F(αθ + (1 − α)θ')
is a Jensen difference divergence.
References
A. Rényi (1961). "On measures of information and entropy". Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561. http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article27.pdf.
A. O. Hero, O.Michael and J. Gorman (2002). Alphadivergences for Classification, Indexing and Retrieval. http://www.eecs.umich.edu/~hero/Preprints/cspl328.pdf.
F. Nielsen and S. Boltz (2010). "The BurbeaRao and Bhattacharyya centroids". arXiv:1004.5049.
Nielsen, Frank; Nock, Richard (2011). "On Rényi and Tsallis entropies and divergences for exponential families". arXiv:1105.3259.
O.A. Rosso EEG analysis using waveletbased information tools. Journal of Neuroscience Methods 153 (2006) 163–182 Rényi entropy as a measure of entanglement in quantum spin chain: F. Franchini, A. R. Its, V. E. Korepin, Journal of Physics A: Math. Theor. 41 (2008) 025302 [1]
T. van Erven (2010). When Data Compression and Statistics Disagree (Ph.D. thesis). hdl:1887/15879 . Chapter 6
See also
 Diversity indices
 Tsallis entropy
 Generalized entropy index
Categories: Information theory
 Entropy and information
Wikimedia Foundation. 2010.
Look at other dictionaries:
Entropy — This article is about entropy in thermodynamics. For entropy in information theory, see Entropy (information theory). For a comparison of entropy in information theory with entropy in thermodynamics, see Entropy in thermodynamics and information… … Wikipedia
Entropy (information theory) — In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information… … Wikipedia
Entropy (disambiguation) — Additional relevant articles may be found in the following categories: Thermodynamic entropy Entropy and information Quantum mechanical entropy Entropy, in thermodynamics, is a measure of the energy in a thermodynamic system not available to do… … Wikipedia
Entropy (general concept) — In many branches of science, entropy refers to a certain measure of the disorder of a system. Entropy is particularly notable as it has a broad, common definition that is shared across physics, mathematics and information science. Although the… … Wikipedia
Minentropy — In probability theory or information theory, the min entropy of a discrete random event x with possible states (or outcomes) 1... n and corresponding probabilities p1... pn is The base of the logarithm is just a scaling constant; for a… … Wikipedia
Binary entropy function — In information theory, the binary entropy function, denoted H(p) , or H {mathrm b}(p) ,, is defined as the entropy of a Bernoulli trial with probability of success p . Mathematically, the Bernoulli trial is modelled as a random variable X that… … Wikipedia
Tsallis entropy — In physics, the Tsallis entropy is a generalization of the standard Boltzmann Gibbs entropy. It was an extension put forward by Constantino Tsallis in 1988. It is defined as:S q(p) = {1 over q 1} left( 1 int p^q(x), dx ight),or in the discrete… … Wikipedia
Alfréd Rényi — (20 March 1921 ndash; 1 February 1970) was a Hungarian mathematician who made contributions in combinatorics and graph theory but mostly in probability theory. [citationtitle=Obituary: Alfred Renyifirst=Davidlast=Kendalljournal=Journal of… … Wikipedia
Alfred Renyi — Alfréd Rényi Alfred Renyi. Alfréd Rényi (20 mars 1921 1er février 1970) était un mathématicien hongrois. Ses contributions sont surtout en combinatoire, en théorie des graphes et en théorie des probabili … Wikipédia en Français
Alfréd Rényi — Alfred Renyi. Alfréd Rényi (20 mars 1921 1er février 1970) était un mathématicien hongrois. Ses contributions sont surtout en combinatoire, en théorie des graphes et en théorie des probabilités. En 1950, il a fondé l Institut de… … Wikipédia en Français