Exact test

Exact test

In statistics, an exact (significance) test is a test where all assumptions upon which the derivation of the distribution of the test statistic is based are met, as opposed to an approximate test, in which the approximation may be made as close as desired by making the sample size big enough. This will result in a significance test that will have a false rejection rate always equal to the significance level of the test. For example an exact test at significance level 5% will in the long run reject true null hypothesis exactly 5% of the time.

Parametric tests, such as those described in exact statistics, are exact tests when the parametric assumptions are fully met, but in practice the use of the term "exact" (significance) "test" is reserved for those tests that do not rest on parametric assumptions – non-parametric tests. However, in practice most implementations of non-parametric test software use asymptotical algorithms for obtaining the significance value, which makes the implementation of the test non-exact.

So when the result of a statistical analysis is said to be an “exact test” or an “exact p-value”, it ought to imply that the test is defined without parametric assumptions and evaluated without using approximate algorithms. In principle however it could also mean that a parametric test has been employed in a situation where all parametric assumptions are fully met, but it is in most cases impossible to prove this completely in a real world situation. Exceptions when it is certain that parametric tests are exact include tests based on the binomial or Poisson distributions. Sometimes permutation test is used as a synonym for exact test, but although all permutation tests are exact tests, not all exact tests are permutation tests.

Definition

The basic equation underlying permutation tests is

:Pr( ext{exact})=sum_{mathbf{y},:,T(mathbf{y})ge T(mathbf{x) Pr(mathbf{y})

where::*x is the outcome actually observed,:*Pr(y) is the probability under the null hypothesis of a potentially observed outcome y,:*"T"(y) is the value of the test statistic for an outcome y, with larger values of "T" representing cases which notionally represent greater departures from the null hypothesis,and where the sum ranges over all outcomes y (including the observed one) that have the same value of the test statistic obtained for the observed sample x, or a larger one .

Example: Pearson's chi-square test versus an exact test

A simple example of the occasion for this concept may be seen by observing that Pearson's chi-square test is an approximate test. Suppose Pearson's chi-square test is used to ascertain whether a six-sided die is "fair", i.e. gives each of the six outcomes equally often. If the die is thrown "n" times, then one "expects" to see each outcome "n"/6 times. The test statistic is

: sum frac{( ext{observed}- ext{expected})^2}{ ext{expected= sum_{k=1}^6 frac{(X_k - n/6)^2}{n/6},

where "X""k" is the number of times outcome "k" is observed. If the null hypothesis of "fairness" is true, then the probability distribution of the test statistic can be made as close as desired to the chi-square distribution with 5 degrees of freedom by making the sample size "n" big enough. But if "n" is small, then the probabilities based on chi-square distributions may not be very close approximations. Finding the exact probability that this test statistic exceeds a certain value then requires combinatorial enumeration of all outcomes of the experiment that result in such a large value of the test statistic. Moreover, it becomes questionable whether the same test statistic ought to be used. A likelihood-ratio test might be preferred as being more powerful, and the test statistic might not be a monotone function of the one above.

Example: Fisher's exact test

Fisher's exact test is exact because the sampling distribution (conditional on the marginals) is known exactly. Compare Pearson's chi-squared test, which (although it tests the same null) is not exact because the distribution of the test statistic is correct only asymptotically.

ee also

* Resampling (statistics)
* Exact statistics

References

Fisher, R. A. 1954 Statistical Methods for research workers. Oliver and Boyd. Mehta, C. R. 1995. SPSS 6.1 Exact test for Windows. Englewood Cliffs, NJ: Prentice Hall.

Mehta, C. R.& Patel, N. R. 1997. Exact inference in categorical data. Biometrics, 53(1), 112-117.

Weerahandi, S. 1995. [http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-40621-3 Exact Statistical Method for Data Analysis.] Springer-Verlag, New York.

Weerahandi, S. 2004. [http://www.wiley-vch.de/publish/en/books/bySubjectST00/bySubSubjectST12/0-471-47017-1/?sID=d05b Generalized Inference in Repeated Measures: Exact Methods in MANOVA and Mixed Models.] Wiley, New York.


Wikimedia Foundation. 2010.

Игры ⚽ Поможем сделать НИР

Look at other dictionaries:

  • exact test — a statistical test based on the actual probability distribution of the data in the study, rather than on an approximation of it …   Medical dictionary

  • Fisher's exact test — is a statistical significance test used in the analysis of categorical data where sample sizes are small. It is named after its inventor, R. A. Fisher, and is one of a class of exact tests. Fisher devised the test following a comment from Muriel… …   Wikipedia

  • Fisher’s exact test — A statistical test of independence much used in medical research. It tests the independence of rows and columns in a 2 X 2 contingency table (with 2 horizontal rows crossing 2 vertical columns creating 4 places for data) based on the exact… …   Medical dictionary

  • Fisher exact test — Fish·er exact test (fishґər) [Sir Ronald Aylmer Fisher, British statistician, 1890–1962] see under test …   Medical dictionary

  • Exact statistics — Exact statistics, such as that described in exact test, is a branch of statistics that was developed to provide more accurate results pertaining to statistical testing and interval estimation by eliminating procedures based on asymptotic and… …   Wikipedia

  • Fisher exact test — a statistical hypothesis test of independence of rows and columns in a 2 Ч 2 contingency table based on the exact sampling distribution of the observed frequencies, useful when any expected value in the table is small …   Medical dictionary

  • Test, Fisher’s exact — A statistical test of independence much used in medical research. It tests the independence of rows and columns in a 2 X 2 contingency table (with 2 horizontal rows crossing 2 vertical columns creating 4 places for data) based on the exact… …   Medical dictionary

  • Test theories of special relativity — are frameworks, which are used for examinations about the validity of Lorentz symmetry and the existence of a preferred frame of reference. A well known example was created by Reza Mansouri and Roman Ulrich Sexl in 1977. [Citation |… …   Wikipedia

  • Test de Khi-2 — Test du χ²  Pour la loi de probabilité, voir Loi du χ². Densité du χ² en fonction du nombre de degrés de liberté Le test du χ² (prononcer …   Wikipédia en Français

  • Test du Chi-2 — Test du χ²  Pour la loi de probabilité, voir Loi du χ². Densité du χ² en fonction du nombre de degrés de liberté Le test du χ² (prononcer …   Wikipédia en Français

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”