Sufficiency (statistics)


Sufficiency (statistics)

In statistics, sufficiency is the property possessed by a statistic, with respect to a parameter, "when no other statistic which can be calculated from the same sample provides any additional information as to the value of the parameter"cite journal
last = Fisher
first = Ronald
authorlink = Ronald Fisher
date =
year = 1922
title = On the Mathematical Foundations Of Theoretical Statistics.
journal = Phil. Trans. R. Soc. Lond. A
volume = 222
pages = 309–68
language = English
doi = 10.1098/rsta.1922.0009
]

This concept was due to Sir Ronald Fisher, and is equivalent to the most general statement of the above that, conditional on the value of a sufficient statistic, the distributions of samples drawn are independent of the underlying parameter(s) the statistic is sufficient for. Both the statistic and the underlying parameter can be vectors.

The concept has fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form, but remains very important in theoretical work.cite journal
last = Stigler
first = Stephen
authorlink = Stephen Stigler
date =
year = 1973
month = Dec.
title = Studies in the History of Probability and Statistics. XXXII: Laplace, Fisher and the Discovery of the Concept of Sufficiency
journal = Biometrika
volume = 60
issue = 3
pages = 439–445
language = English
doi = 10.2307/2334992
]

Mathematical definition

The concept is most general when defined as follows: a statistic "T"("X") is sufficient for underlying parameter "θ" precisely if the conditional probability distribution of the data "X", given the statistic "T"("X"), is independent of the parameter "θ",cite book | last = Casella | first = George | coauthors = Berger, Roger L. | title = Statistical Inference, 2nd ed | publisher=Duxbury Press | date = 2002] i.e.

:Pr(X=x|T(X)=t, heta) = Pr(X=x|T(X)=t), ,or in shorthand:Pr(x|t, heta) = Pr(x|t).,

Example

As an example, the sample mean is sufficient for the mean (μ) of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself.

Fisher-Neyman factorization theorem

"Fisher's factorization theorem" or "factorization criterion" provides a convenient characterization of a sufficient statistic. If the probability density function is ƒ"θ"("x"), then "T" is sufficient for "θ" if and only if functions "g" and "h" can be found such that

: f_ heta(x)=h(x) , g_ heta(T(x)), ,!

i.e. the density ƒ can be factored into a product such that one factor, "h", does not depend on "θ" and the other factor, which does depend on "θ", depends on "x" only through "T"("x").

Interpretation

An implication of the theorem is that when using likelihood-based inference, two sets of data yielding the same value for the sufficient statistic "T"("X") will always yield the same inferences about θ. By the factorization criterion, the likelihood's dependence on θ is only in conjunction with "T"("X"). As this is the same in both cases, the dependence on θ will be the same as well, leading to identical inferences.

Proof for the continuous case

Due to Hogg and Craig (ISBN 978-0023557224). Let "X"1, "X"2, ..., "X"n, denote a random sample from a distribution having the pdf "f"("x",&theta;) for &gamma; < &theta; < &delta;. Let "Y" = "u"("X"1, "X"2, ..., "X"n) be a statistic whose pdf is "g"("y";&theta;). Then "Y" = "u"("X"1, "X"2, ..., "X"n) is a sufficient statistic for &theta; if and only if, for some function "H",

: prod_{i=1}^{n} f(x_i; heta) = g left [u(x_1, x_2, dots, x_n); heta ight] H(x_1, x_2, dots, x_n). ,!

First, suppose that: prod_{i=1}^{n} f(x_i; heta) = g left [u(x_1, x_2, dots, x_n); heta ight] H(x_1, x_2, dots, x_n). ,!

We shall make the transformation "y""i" = "u"i("x"1, "x"2, ..., "x""n"), for "i" = 1, ..., "n", having inverse functions "x"i = "w"i("y"1, "y"2, ..., "y""n"), for "i" = 1, ..., "n", and Jacobian "J". Thus,

: prod_{i=1}^{n} f left [ w_i(y_1, y_2, dots, y_n); heta ight] =
J| g(y; heta) H left [ w_1(y_1, y_2, dots, y_n), dots, w_n(y_1, y_2, dots, y_n) ight] .

The left-hand member is the joint pdf "g"("y"1, "y"2, ..., "y""n"; &theta;) of "Y"1 = "u"1("X"1, ..., "X"n), ..., "Y"n = "u"n("X"1, ..., "X""n"). In the right-hand member, g(y_1,dots,y_n; heta) is the pdf of Y_1, so that H [ w_1, dots , w_n] |J| is the quotient of g(y_1,dots,y_n; heta) and g_1(y_1; heta); that is, it is the conditional pdf h(y_2, dots, y_n | y_1; heta) of Y_2,dots,Y_n given Y_1=y_1.

But H(x_1,x_2,dots,x_n), and thus Hleft [w_1(y_1,dots,y_n), dots, w_n(y_1, dots, y_n)) ight] , was given not to depend upon heta. Since heta was not introduced in the transformation and accordingly not in the Jacobian J, it follows that h(y_2, dots, y_n | y_1; heta) does not depend upon heta and that Y_1 is a sufficient statistics for heta.

The converse is proven by taking:

:g(y_1,dots,y_n; heta)=g_1(y_1; heta) h(y_2, dots, y_n | y_1),,

where h(y_2, dots, y_n | y_1) does not depend upon heta because Y_2 ... Y_n depend only upon X_1 ... X_n which are independent on Theta when conditioned by Y_1, a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian J, and replace y_1, dots, y_n by the functions u_1(x_1, dots, x_n), dots, u_n(x_1,dots, x_n) in x_1,dots, x_n. This yields

:frac{gleft [ u_1(x_1, dots, x_n), dots, u_n(x_1, dots, x_n); heta ight] }

is a function that does not depend upon heta.

Proof for the discrete case

We use the shorthand notation to denote the joint probability of (X, T(X)) by f_ heta(x,t). Since T is a function of X, we have f_ heta(x,t) = f_ heta(x) and thus:

:f_ heta(x) = f_ heta(x,t) = f_{ heta | t}(x) f_ heta(t)

with the last equality being true by the definition of conditional probability distributions. Thus f_ heta(x)=a(x) b_ heta(t) with a(x) = f_{ heta | t}(x) and b(x) = f_ heta(t).

Reciprocally, if f_ heta(x)=a(x) b_ heta(t), we have

:egin{align}f_ heta(t) & = sum _{x : T(x) = t} f_ heta(x, t) \ & = sum _{x : T(x) = t} f_ heta(x) \& = sum _{x : T(x) = t} a(x) b_ heta(t) \& = left( sum _{x : T(x) = t} a(x) ight) b_ heta(t)end{align}

With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over t.

Thus, the conditional probability distribution is::egin{align}f_{ heta|t}(x)& = frac{f_ heta(x, t)}{f_ heta(t)} \& = frac{f_ heta(x)}{f_ heta(t)} \& = frac{a(x) b_ heta(t)}{left( sum _{x : T(x) = t} a(x) ight) b_ heta(t)} \& = frac{a(x)}{sum _{x : T(x) = t} a(x)}end{align}

With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on heta and thus T is a sufficient statistic. [cite web | url=http://cnx.org/content/m11480/1.6/ | title=The Fisher-Neyman Factorization Theorem]

Minimal sufficiency

A sufficient statistic is minimal sufficient if it can be represented as a function of any other sufficient statistic. In other words, "S"("X") is minimal sufficient if and only if
#"S"("X") is sufficient, and
#if "T"("X") is sufficient, then there exists a function "f" such that "S"("X") = "f"("T"("X")).

Intuitively, a minimal sufficient statistic "most efficiently" captures all possible information about the parameter "&theta;".

A useful characterization of minimal sufficiency is that when the density "f"&theta; exists, "S"("X") is minimal sufficient if and only if:frac{f_ heta(x)}{f_ heta(y)} is independent of "&theta;" :Longleftrightarrow "S"("x") = "S"("y")

This follows as a direct consequence from the Fisher's factorization theorem stated above.

A sufficient and complete statistic is necessarily minimal sufficient. A minimal sufficient statistic always exists; a complete statistic need not exist.

Examples

Bernoulli distribution

If "X"1, ...., "X""n" are independent Bernoulli-distributed random variables with expected value "p", then the sum "T"("X") = "X"1 + ... + "X""n" is a sufficient statistic for "p" (here 'success' corresponds to X_i=1 and 'failure' to X_i=0; so "T" is the total number of successes)

This is seen by considering the joint probability distribution:

: Pr(X=x)=P(X_1=x_1,X_2=x_2,ldots,X_n=x_n).

Because the observations are independent, this can be written as

:p^{x_1}(1-p)^{1-x_1} p^{x_2}(1-p)^{1-x_2}cdots p^{x_n}(1-p)^{1-x_n} ,!

and, collecting powers of "p" and 1 − "p", gives

:p^{sum x_i}(1-p)^{n-sum x_i}=p^{T(x)}(1-p)^{n-T(x)} ,!

which satisfies the factorization criterion, with "h"("x")=1 being just a constant.

Note the crucial feature: the unknown parameter "p" interacts with the data "x" only via the statistic "T"("x") = &Sigma; "x""i".

Uniform distribution

If "X"1, ...., "X""n" are independent and uniformly distributed on the interval [0,&theta;] , then "T"("X") = max("X"1, ...., "X""n" ) is sufficient for &theta;.

To see this, consider the joint probability distribution:

:Pr(X=x)=P(X_1=x_1,X_2=x_2,ldots,X_n=x_n).

Because the observations are independent, this can be written as

:frac{operatorname{H}( heta-x_1)}{ heta}cdotfrac{operatorname{H}( heta-x_2)}{ heta}cdot,cdots,cdotfrac{operatorname{H}( heta-x_n)}{ heta} ,!

where H("x") is the Heaviside step function. This may be written as

:frac{operatorname{H}left( heta-max_i {,x_i,} ight)}{ heta^n},!

which can be viewed as a function of only "&theta;" and maxi("X"i) = "T"("X"). This shows that the factorization criterion is satisfied, again where "h"("x")=1 is constant. Note that the parameter &theta; interacts with the data only through the data's maximum.

Poisson distribution

If "X"1, ...., "X""n" are independent and have a Poisson distribution with parameter "&lambda;", then the sum "T"("X") = "X"1 + ... + "X""n" is a sufficient statistic for "&lambda;".

To see this, consider the joint probability distribution:

:Pr(X=x)=P(X_1=x_1,X_2=x_2,ldots,X_n=x_n).

Because the observations are independent, this can be written as

:{e^{-lambda} lambda^{x_1} over x_1 !} cdot {e^{-lambda} lambda^{x_2} over x_2 !} cdot,cdots,cdot {e^{-lambda} lambda^{x_n} over x_n !} ,!

which may be written as

:e^{-nlambda} lambda^{(x_1+x_2+cdots+x_n)} cdot {1 over x_1 ! x_2 !cdots x_n ! } ,!

which shows that the factorization criterion is satisfied, where "h"("x") is the reciprocal of the product of the factorials. Note the parameter &lambda; interacts with the data only through its sum "T"("X").

Rao-Blackwell theorem

Sufficiency finds a useful application in the Rao-Blackwell theorem. It states that if "g"("X") is any kind of estimator of "&theta;", then typically the conditional expectation of "g"("X") given "T"("X") is a better estimator of "&theta;", and is never worse. Sometimes one can very easily construct a very crude estimator "g"("X"), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.

ee also

*Completeness of a statistic
*Basu's theorem on independence of complete sufficient and ancillary statistics
*Lehman-Scheffe theorem: a complete sufficient estimator is the best estimator of its expectation

References


Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Sufficiency — may refer to: * Logical sufficiency; see necessary and sufficient conditions * sufficiency (statistics), sufficiency in statistical inference;See also * Self sufficiency * Eco sufficiency …   Wikipedia

  • List of statistics topics — Please add any Wikipedia articles related to statistics that are not already on this list.The Related changes link in the margin of this page (below search) leads to a list of the most recent changes to the articles listed below. To see the most… …   Wikipedia

  • Glossary of probability and statistics — The following is a glossary of terms. It is not intended to be all inclusive. Concerned fields *Probability theory *Algebra of random variables (linear algebra) *Statistics *Measure theory *Estimation theory Glossary *Atomic event : another name… …   Wikipedia

  • List of mathematics articles (S) — NOTOC S S duality S matrix S plane S transform S unit S.O.S. Mathematics SA subgroup Saccheri quadrilateral Sacks spiral Sacred geometry Saddle node bifurcation Saddle point Saddle surface Sadleirian Professor of Pure Mathematics Safe prime Safe… …   Wikipedia

  • Estimation theory — is a branch of statistics and signal processing that deals with estimating the values of parameters based on measured/empirical data. The parameters describe an underlying physical setting in such a way that the value of the parameters affects… …   Wikipedia

  • Sufficient statistic — In statistics, a sufficient statistic is a statistic which has the property of sufficiency with respect to a statistical model and its associated unknown parameter, meaning that no other statistic which can be calculated from the same sample… …   Wikipedia

  • Amtrak — For other uses, see Amtrak (disambiguation). Amtrak Map of the Amtrak system (2007) Reporting mark …   Wikipedia

  • Exponential family — Not to be confused with the exponential distribution. Natural parameter links here. For the usage of this term in differential geometry, see differential geometry of curves. In probability and statistics, an exponential family is an important… …   Wikipedia

  • Economy of North Korea — Economic centre of North Korea Currency North Korean won Fiscal year …   Wikipedia

  • M. S. Bartlett — Born 18 June 1910(1910 06 18) Died 8 January 2002(2002 01 08) (aged 91) …   Wikipedia


Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”

We are using cookies for the best presentation of our site. Continuing to use this site, you agree with this.