# Design of experiments

﻿
Design of experiments

In general usage, design of experiments (DOE) or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. However, in statistics, these terms are usually used for controlled experiments. Other types of study, and their design, are discussed in the articles on opinion polls and statistical surveys (which are types of observational study), natural experiments and quasi-experiments (for example, quasi-experimental design). See Experiment for the distinction between these types of experiments or studies.

In the design of experiments, the experimenter is often interested in the effect of some process or intervention (the "treatment") on some objects (the "experimental units"), which may be people, parts of people, groups of people, plants, animals, materials, etc. Design of experiments is thus a discipline that has very broad application across all the natural and social sciences.

## History of development

### Controlled experimentation on scurvy

In 1747, while serving as surgeon on HMS Salisbury, James Lind carried out a controlled experiment to develop a cure for scurvy.[1]

Lind selected 12 men from the ship, all suffering from scurvy. Lind limited his subjects to men who "were as similar as I could have them", that is he provided strict entry requirements to reduce extraneous variation. He divided them into six pairs, giving each pair different supplements to their basic diet for two weeks. The treatments were all remedies that had been proposed:

• A quart of cider every day
• Twenty five gutts (drops) of elixir vitriol (sulphuric acid) three times a day upon an empty stomach,
• One half-pint of seawater every day
• A mixture of garlic, mustard, and horseradish in a lump the size of a nutmeg
• Two spoonfuls of vinegar three times a day
• Two oranges and one lemon every day.

The men who had been given citrus fruits recovered dramatically within a week. One of them returned to duty after 6 days and the other cared for the rest. The others experienced some improvement, but nothing was comparable to the citrus fruits, which were proved to be substantially superior to the other treatments.

### Statistical experiments, following Charles S. Peirce

A theory of statistical inference was developed by Charles S. Peirce in "Illustrations of the Logic of Science" (1877–1878) and "A Theory of Probable Inference" (1883), two publications that emphasized the importance of randomization-based inference in statistics.

#### Randomized experiments

Charles S. Peirce randomly assigned volunteers to a blinded, repeated-measures design to evaluate their ability to discriminate weights.[2][3][4][5] Peirce's experiment inspired other researchers in psychology and education, which developed a research tradition of randomized experiments in laboratories and specialized textbooks in the 1800s.[2][3][4][5]

#### Optimal designs for regression models

Charles S. Peirce also contributed the first English-language publication on an optimal design for regression-models in 1876.[6] A pioneering optimal design for polynomial regression was suggested by Gergonne in 1815. In 1918 Kirstine Smith published optimal designs for polynomials of degree six (and less).

### Sequences of experiments

The use of a sequence of experiments, where the design of each may depend on the results of previous experiments, including the possible decision to stop experimenting, is within the scope of Sequential analysis, a field that was pioneered[7] by Abraham Wald in the context of sequential tests of statistical hypotheses.[8] Herman Chernoff wrote an overview of optimal sequential designs,[9] while adaptive designs have been surveyed by S. Zacks.[10] One specific type of sequential design is the "two-armed bandit", generalized to the multi-armed bandit, on which early work was done by Herbert Robbins in 1952.[11]

## Principles of experimental design, following Ronald A. Fisher

A methodology for designing experiments was proposed by Ronald A. Fisher, in his innovative book The Design of Experiments (1935). As an example, he described how to test the hypothesis that a certain lady could distinguish by flavour alone whether the milk or the tea was first placed in the cup. While this sounds like a frivolous application, it allowed him to illustrate the most important ideas of experimental design:

Comparison
In many fields of study it is hard to reproduce measured results exactly. Comparisons between treatments are much more reproducible and are usually preferable. Often one compares against a standard, scientific control, or traditional treatment that acts as baseline.
Randomization

Random assignment is the process of assigning individuals at random to groups or to different groups in an experiment. The random assignment of individuals to groups (or conditions within a group) distinguishes a rigorous, "true" experiment from an adequate, but less-than-rigorous, "quasi-experiment".[12]

There is an extensive body of mathematical theory that explores the consequences of making the allocation of units to treatments by means of some random mechanism such as tables of random numbers, or the use of randomization devices such as playing cards or dice. Provided the sample size is adequate, the risks associated with random allocation (such as failing to obtain a representative sample in a survey, or having a serious imbalance in a key characteristic between a treatment group and a control group) are calculable and hence can be managed down to an acceptable level. Random does not mean haphazard, and great care must be taken that appropriate random methods are used.
Replication
Measurements are usually subject to variation and uncertainty. Measurements are repeated and full experiments are replicated to help identify the sources of variation, to better estimate the true effects of treatments, to further strengthen the experiment's reliability and validity, and to add to the existing knowledge of about the topic. [13] However, certain conditions must be met before the replication of the experiment is commenced: the original research question has been published in a peer-reviewed journal or widely cited, the researcher is independent of the original experiment, the researcher must first try to replicate the original findings using the original data, and the write-up should state that the study conducted is a replication study that tried to follow the original study as strictly as possible. [14]
Blocking
Blocking is the arrangement of experimental units into groups (blocks) consisting of units that are similar to one another. Blocking reduces known but irrelevant sources of variation between units and thus allows greater precision in the estimation of the source of variation under study.
Orthogonality
Example of orthogonal factorial design
Orthogonality concerns the forms of comparison (contrasts) that can be legitimately and efficiently carried out. Contrasts can be represented by vectors and sets of orthogonal contrasts are uncorrelated and independently distributed if the data are normal. Because of this independence, each orthogonal treatment provides different information to the others. If there are T treatments and T – 1 orthogonal contrasts, all the information that can be captured from the experiment is obtainable from the set of contrasts.
Factorial experiments
Use of factorial experiments instead of the one-factor-at-a-time method. These are efficient at evaluating the effects and possible interactions of several factors (independent variables).

Analysis of the design of experiments was built on the foundation of the analysis of variance, a collection of models in which the observed variance is partitioned into components due to different factors which are estimated and/or tested.

## Example

This example is attributed to Harold Hotelling.[9] It conveys some of the flavor of those aspects of the subject that involve combinatorial designs.

The weights of eight objects are to be measured using a pan balance and set of standard weights. Each weighing measures the weight difference between objects placed in the left pan vs. any objects placed in the right pan by adding calibrated weights to the lighter pan until the balance is in equilibrium. Each measurement has a random error. The average error is zero; the standard deviations of the probability distribution of the errors is the same number σ on different weighings; and errors on different weighings are independent. Denote the true weights by

$\theta_1, \dots, \theta_8.\,$

We consider two different experiments:

1. Weigh each object in one pan, with the other pan empty. Let Xi be the measured weight of the ith object, for i = 1, ..., 8.
2. Do the eight weighings according to the following schedule and let Yi be the measured difference for i = 1, ..., 8:
$\begin{matrix} & \mbox{left pan} & \mbox{right pan} \\ \mbox{1st weighing:} & 1\ 2\ 3\ 4\ 5\ 6\ 7\ 8 & \text{(empty)} \\ \mbox{2nd:} & 1\ 2\ 3\ 8\ & 4\ 5\ 6\ 7 \\ \mbox{3rd:} & 1\ 4\ 5\ 8\ & 2\ 3\ 6\ 7 \\ \mbox{4th:} & 1\ 6\ 7\ 8\ & 2\ 3\ 4\ 5 \\ \mbox{5th:} & 2\ 4\ 6\ 8\ & 1\ 3\ 5\ 7 \\ \mbox{6th:} & 2\ 5\ 7\ 8\ & 1\ 3\ 4\ 6 \\ \mbox{7th:} & 3\ 4\ 7\ 8\ & 1\ 2\ 5\ 6 \\ \mbox{8th:} & 3\ 5\ 6\ 8\ & 1\ 2\ 4\ 7 \end{matrix}$
Then the estimated value of the weight θ1 is
$\widehat{\theta}_1 = \frac{Y_1 + Y_2 + Y_3 + Y_4 - Y_5 - Y_6 - Y_7 - Y_8}{8}.$
Similar estimates can be found for the weights of the other items. For example
$\widehat{\theta}_2 = \frac{Y_1 + Y_2 - Y_3 - Y_4 + Y_5 + Y_6 - Y_7 - Y_8}{8}.$

The question of design of experiments is: which experiment is better?

The variance of the estimate X1 of θ1 is σ2 if we use the first experiment. But if we use the second experiment, the variance of the estimate given above is σ2/8. Thus the second experiment gives us 8 times as much precision for the estimate of a single item, and estimates all items simultaneously, with the same precision. What is achieved with 8 weighings in the second experiment would require 64 weighings if items are weighed separately. However, note that the estimates for the items obtained in the second experiment have errors which are correlated with each other.

Many problems of the design of experiments involve combinatorial designs, as in this example.

## Statistical control

It is best for a process to be in reasonable statistical control prior to conducting designed experiments. When this is not possible, proper blocking, replication, and randomization allow for the careful conduct of designed experiments.[15] To control for nuisance variables, researchers institute control checks as additional measures. Investigators should ensure that uncontrolled influences (e.g., source credibility perception) are measured do not skew the findings of the study. A manipulation check is one example of a control check. Manipulation checks allow investigators to isolate the chief variables to strengthen support that these variables are operating as planned.

## Experimental designs after Fisher

Some efficient designs for estimating several main effects simultaneously were found by Raj Chandra Bose and K. Kishen in 1940 at the Indian Statistical Institute, but remained little known until the Plackett-Burman designs were published in Biometrika in 1946. About the same time, C. R. Rao introduced the concepts of orthogonal arrays as experimental designs. This was a concept which played a central role in the development of Taguchi methods by Genichi Taguchi, which took place during his visit to Indian Statistical Institute in early 1950s. His methods were successfully applied and adopted by Japanese and Indian industries and subsequently were also embraced by US industry albeit with some reservations.

In 1950, Gertrude Mary Cox and William Gemmell Cochran published the book Experimental Designs which became the major reference work on the design of experiments for statisticians for years afterwards.

Developments of the theory of linear models have encompassed and surpassed the cases that concerned early writers. Today, the theory rests on advanced topics in linear algebra, algebra and combinatorics.

As with other branches of statistics, experimental design is pursued using both frequentist and Bayesian approaches: In evaluating statistical procedures like experimental designs, frequentist statistics studies the sampling distribution while Bayesian statistics updates a probability distribution on the parameter space.

Some important contributors to the field of experimental designs are C. S. Peirce, R. A. Fisher, F. Yates, C. R. Rao, R. C. Bose, J. N. Srivastava, Shrikhande S. S., D. Raghavarao, W. G. Cochran, O. Kempthorne, W. T. Federer, A. S. Hedayat, J. A. Nelder, R. A. Bailey, J. Kiefer, W. J. Studden, F. Pukelsheim, D. R. Cox, H. P. Wynn, A. C. Atkinson, G. E. P. Box and G. Taguchi. The textbooks of D. Montgomery and R. Myers have reached generations of students and practitioners.

## Notes

1. ^ Dunn, Peter (January 1997). "James Lind (1716-94) of Edinburgh and the treatment of scurvy". Archive of Disease in Childhood Foetal Neonatal (United Kingdom: British Medical Journal Publishing Group) 76 (1): 64–65. doi:10.1136/fn.76.1.F64. PMC 1720613. PMID 9059193. Retrieved 2009-01-17.
2. ^ a b Peirce, Charles Sanders; Jastrow, Joseph (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences 3: 73–83.
3. ^ a b Hacking, Ian (September 1988). "Telepathy: Origins of Randomization in Experimental Design". Isis 79 (3): 427–451. doi:10.1086/354775. JSTOR 234674. MR1013489.
4. ^ a b Stephen M. Stigler (November 1992). "A Historical View of Statistical Concepts in Psychology and Educational Research". American Journal of Education 101 (1): 60–70. doi:10.1086/444032. JSTOR 1085417.
5. ^ a b Trudy Dehue (December 1997). "Deception, Efficiency, and Random Groups: Psychology and the Gradual Origination of the Random Group Design". Isis 88 (4): 653–673. doi:10.1086/383850. PMID 9519574.
6. ^ Peirce, C. S. (1876). "Note on the Theory of the Economy of Research". Coast Survey Report: pp. 197–201. , actually published 1879, NOAA PDF Eprint.
Reprinted in Collected Papers 7, paragraphs 139–157, also in Writings 4, pp. 72–78, and in Peirce, C. S. (July–August 1967). "Note on the Theory of the Economy of Research". Operations Research 15 (4): 643–648. doi:10.1287/opre.15.4.643. JSTOR 168276.
7. ^ Johnson, N.L. (1961). "Sequential analysis: a survey." Journal of the Royal Statistical Society, Series A. Vol. 124 (3), 372–411. (pages 375–376)
8. ^ Wald, A. (1945) "Sequential Tests of Statistical Hypotheses", Annals of Mathematical Statistics, 16 (2), 117–186.
9. ^ a b Herman Chernoff, Sequential Analysis and Optimal Design, SIAM Monograph, 1972.
10. ^ Zacks, S. (1996) "Adaptive Designs for Parametric Models". In: Ghosh, S. and Rao, C. R., (Eds) (1996). "Design and Analysis of Experiments," Handbook of Statistics, Volume 13. North-Holland. ISBN 0-444-82061-2. (pages 151–180)
11. ^ Robbins, H. (1952). "Some Aspects of the Sequential Design of Experiments". Bulletin of the American Mathematical Society 58: 527–535. doi:10.1090/S0002-9904-1952-09620-8.
12. ^ Creswell, J.W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (3rd). Upper Saddle River, NJ: Prentice Hall. 2008, p. 300. ISBN: 10 0136135501
13. ^ Dr. Hani (2009). "Replication study". Retrieved 27 October 2011.
14. ^ Burman, Leonard E. (2010). "A call for replication studies" (journal article). Public Finance Review. pp. 787-793. doi:10.1177/1091142110385210. Retrieved 27 October 2011.
15. ^ Bisgaard, S (2008) "Must a Process be in Statistical Control before Conducting Designed Experiments?", Quality Engineering, ASQ, 20 (2), pp 143 - 176

## References

• Peirce, C. S. (1877–1878), "Illustrations of the Logic of Science" (series), Popular Science Monthly, vols. 12-13. Relevant individual papers:
• (1878 March), "The Doctrine of Chances", Popular Science Monthly, v. 12, March issue, pp. 604–615. Internet Archive Eprint.
• (1878 April), "The Probability of Induction", Popular Science Monthly, v. 12, pp. 705–718. Internet Archive Eprint.
• (1878 June), "The Order of Nature", Popular Science Monthly, v. 13, pp. 203–217.Internet Archive Eprint.
• (1878 August), "Deduction, Induction, and Hypothesis", Popular Science Monthly, v. 13, pp. 470–482. Internet Archive Eprint.
• Peirce, C. S. (1883), "A Theory of Probable Inference", Studies in Logic, pp. 126-181, Little, Brown, and Company. (Reprinted 1983, John Benjamins Publishing Company, ISBN 9027232717)

Wikimedia Foundation. 2010.

### Look at other dictionaries:

• Design of Experiments — Die statistische Versuchsplanung (englisch design of experiments, DOE) wird bei Entwicklung und Optimierung von Produkten oder Prozessen eingesetzt. Da Versuche Ressourcen benötigen (Personal, Zeit, Geräte usw.), sieht sich der… …   Deutsch Wikipedia

• Design of Experiments — n. DOE, plan or model prepared and followed when gathering data in an experiment …   English contemporary dictionary

• Multifactor design of experiments software — Software that is used for designing factorial experiments plays an important role in scientific experiments generally and represents a route to the implementation of design of experiments procedures that derive from statistical and combinatoric… …   Wikipedia

• The Design of Experiments — is a 1935 book by the British statistician R.A. Fisher, which effectively founded the field of experimental design …   Wikipedia

• Design Expert — Entwickler Stat Ease, Inc. Aktuelle Version 8.0.1 (2010) Betriebssystem Windows Kategorie Statistik Software Lizenz …   Deutsch Wikipedia

• Design for Six Sigma — (DFSS) is a separate and emerging business process management methodology related to traditional Six Sigma. While the tools and order used in Six Sigma require a process to be in place and functioning, DFSS has the objective of determining the… …   Wikipedia

• Design theory — can refer to any theory relating to design in general. Design theory may also refer to: Engineering and industrial design C K theory Design science C K theory Mathematics Combinatorial design Block design Symmetric design Design of experiments… …   Wikipedia

• Design of quasi-experiments — The design of a quasi experiment relates to a particular type of experiment or other study in which one has little or no control over the allocation of the treatments or other factors being studied. The key difference in this empirical approach… …   Wikipedia

• Design effect — In statistics, the design effect is an adjustment used in some kinds of studies, such as cluster randomised trials, to allow for the design structure. The adjustment inflates the variance of parameter estimates, and therefore their standard… …   Wikipedia

• Experiments in Art and Technology — (E.A.T.) was a non profit and tax exempt organization established to develop collaborations between artists and engineers. E.A.T. initiated and carried out projects that expanded the role of the artist in contemporary society and helped eliminate …   Wikipedia