Experimental Method

Concept

The experimental method is the research strategy in which the investigator deliberately manipulates one or more independent variables, controls other influences, and observes the effect on a dependent variable. Because the researcher decides who receives the treatment, it is the strongest method for establishing cause and effect. A true experiment has three hallmarks: manipulation of the independent variable, control (usually through a control group and constancy of conditions) and randomisation. R. A. Fisher laid the statistical foundations in the 1920s–30s with the principles of randomisation, replication and local control (blocking), and developed analysis of variance. J. S. Mill’s canons (agreement, difference, concomitant variation and residues) are the older logical ancestors of experimental reasoning. Experiments may run in a laboratory (high control) or in the field (high realism).

Concept

The control group gives the baseline against which the effect of the treatment is judged. For that comparison to be fair, the groups must be equivalent at the start. Random assignment gives every participant an equal chance of being placed in any group, so that known and unknown differences are spread out evenly on average; this is the main safeguard of internal validity. Do not confuse it with random selection, which is the way a sample is drawn from the population and supports external validity. When randomisation is impossible or the sample is small, matching pairs participants who are alike on a key variable such as IQ or age and then splits each pair between groups. Matching controls only the variables that were matched, which is its weakness. Counterbalancing varies the order of conditions across participants to control order effects in repeated-measures designs.

Concept

The placebo effect is an improvement caused by a participant’s expectation rather than by the actual treatment, so many medical and psychological experiments give the control group an inert substance or sham procedure. Blinding prevents expectations from distorting results. In a single-blind design the participants do not know which group they are in; in a double-blind design neither the participants nor the experimenters who deal with them know; in a triple-blind design the data analysts are also unaware. Double-blinding guards against demand characteristics (participants guessing what is wanted) and experimenter bias (the Rosenthal or Pygmalion effect, in which the researcher’s expectations unintentionally influence outcomes).

Concept

Following Campbell and Stanley (1963), internal validity is the extent to which the observed change in the dependent variable can really be attributed to the independent variable rather than to something else. External validity is the extent to which the results can be generalised to other persons (population validity), places (ecological validity) and times. Cook and Campbell (1979) added construct validity (are the concepts correctly operationalised) and statistical conclusion validity (are the statistical inferences sound). There is a trade-off: tight laboratory control raises internal validity but may lower external validity, whereas field studies do the reverse.

Concept

Threats to internal validity are rival explanations for the observed effect. History is an outside event occurring between pretest and posttest. Maturation is natural change in participants with time (growing older, tired or wiser). Testing is the effect of taking a pretest on the later score. Instrumentation is a change in the measuring tool or the observers. Statistical regression is the tendency of extreme scorers to move toward the mean on retesting. Selection means groups differ from the start. Mortality (attrition) is the loss of participants, especially when dropouts differ between groups. Threats to external validity include pretest sensitisation, interaction of selection and treatment, reactive arrangements such as the Hawthorne effect (behaviour changes because people know they are being studied, from the Western Electric studies associated with Elton Mayo) and multiple-treatment interference.

Concept

Pre-experimental designs lack control, for instance the one-shot case study (X then O), the one-group pretest–posttest (O X O) and the static-group comparison; they cannot rule out many threats. True experimental designs use random assignment: the posttest-only control group design (R X O / R O), the pretest–posttest control group design (R O X O / R O O) and the Solomon four-group design (Solomon, 1949), which combines both to detect the effect of pretesting. Quasi-experimental designs, such as the non-equivalent control group design or the time-series design, compare groups without random assignment. Factorial designs study two or more independent variables at once and reveal interaction effects; a 2 × 3 design has two factors with 2 and 3 levels. A randomised block design controls one nuisance variable, and a Latin square controls two by arranging treatments so each appears once in every row and column. Within-subjects (repeated-measures) designs expose the same people to all conditions, while between-subjects designs use different people for each condition.