- Entropy rate
The entropy rate of a
stochastic processis, informally, the time density of the average information in a stochastic process. For stochastic processes with a countableindex, the entropy rate is the limit of the joint entropyof members of the process divided by , as tends to infinity:
when the limit exists. An alternative, related quantity is:
strongly stationarystochastic processes, .
Entropy rates for Markov chains
Since a stochastic process defined by a
Markov chainwhich is irreducibleand aperiodichas a stationary distribution, the entropy rate is independent of the initial distribution.
For example, for such a Markov chain "Y""k" defined on a
countablenumber of states, given the transition matrix"P""ij", "H"("Y") is given by:
where "μ""i" is the
stationary distributionof the chain.
A simple consequence of this definition is that the entropy rate of an i.i.d.
stochastic processhas an entropy rate that is the same as the entropyof any individual member of the process.
Information source (mathematics)
Markov information source
* Cover, T. and Thomas, J. (1991) Elements of Information Theory, John Wiley and Sons, Inc., ISBN 0471062596 [http://www3.interscience.wiley.com/cgi-bin/bookhome/110438582?CRETRY=1&SRETRY=0]
* [http://www.eng.ox.ac.uk/samp Systems Analysis, Modelling and Prediction (SAMP), University of Oxford]
MATLABcode for estimating information-theoretic quantities for stochastic processes.
Wikimedia Foundation. 2010.
Look at other dictionaries:
Entropy (information theory) — In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information… … Wikipedia
Entropy — This article is about entropy in thermodynamics. For entropy in information theory, see Entropy (information theory). For a comparison of entropy in information theory with entropy in thermodynamics, see Entropy in thermodynamics and information… … Wikipedia
Rate–distortion theory — is a major branch of information theory which provides the theoretical foundations for lossy data compression; it addresses the problem of determining the minimal amount of entropy (or information) R that should be communicated over a channel, so … Wikipedia
Rate–distortion optimization — (also RDO or RD) is a method of improving video quality in video compression. The name refers to the optimization of the amount of distortion (loss of video quality) against the amount of data required to encode the video, the rate . While it is… … Wikipedia
Entropy encoding — In information theory an entropy encoding is a lossless data compression scheme that is independent of the specific characteristics of the medium. One of the main types of entropy coding creates and assigns a unique prefix free code to each… … Wikipedia
Entropy (disambiguation) — Additional relevant articles may be found in the following categories: Thermodynamic entropy Entropy and information Quantum mechanical entropy Entropy, in thermodynamics, is a measure of the energy in a thermodynamic system not available to do… … Wikipedia
Entropy (general concept) — In many branches of science, entropy refers to a certain measure of the disorder of a system. Entropy is particularly notable as it has a broad, common definition that is shared across physics, mathematics and information science. Although the… … Wikipedia
Entropy (anesthesiology) — In anesthesiology, entropy is a neurophysiologic monitor of the patient s cerebral cortical function, designed to express the likelihood of consciousness, intraoperative awareness, or awareness with recall during a surgical procedure.Other Vital… … Wikipedia
entropy — [ ɛntrəpi] noun 1》 Physics a thermodynamic quantity representing the unavailability of a system s thermal energy for conversion into mechanical work, often interpreted as the degree of disorder or randomness in the system. 2》 (in information… … English new terms dictionary
entropy — n. 1 Physics a measure of the unavailability of a system s thermal energy for conversion into mechanical work. 2 Physics a measure of the disorganization or degradation of the universe. 3 a measure of the rate of transfer of information in a… … Useful english dictionary