What is an experiment?

The other day the meaning of the term "experiment" was called into question. It matters, because a student and I have a paper in press in which we use the emphasis placed on experimentation to distinguish between two schools of thought in Adaptive Management. I don't want to pre-empt Jamie's paper here, but I did want to talk about what I think an experiment is.
Well, according to Wikipedia, an experiment is "...the step in the scientific method that arbitrates between competing models or hypotheses." (Aside: it is interesting that there is a distinction made between model and hypothesis - for another time perhaps.) OK, I can't see anything wrong with that, but we need to dig a little bit deeper. There are two additional attributes that are important for distinguishing between methods of arbitrating among hypotheses: the number of simultaneous experimental treatments and the amount of replication within treatments.

The number of simultaneous experimental treatments is fairly obvious - how many different manipulations of the system under study are in use? This could range from one (an observational study of existing conditions) to many (a laboratory study with positive and negative control treatments and a dose response). Is the term "experiment" appropriate across this entire range?

The second attribute is the amount of replication within a treatment - in how many different places and times was the effect of the treatment observed? This too can range from one to many.

I believe that the term experiment is appropriate when the number of simultaneous experimental treatments is greater than one, regardless of how much replication is present. Replication does matter, but it doesn't affect the ability of the experimenter to determine causation. Rather it affects the scope of the causation - with only one replicate per treatment it is not possible to generalize beyond the set of objects studied. Within that set it is still possible to determine if a hypothesis is consistent with the data, and attribute the differences between treatment responses to the manipulation.

This attribution of causation is the reason for the treatments to be "simultaneous", because this reduces the extent of unmeasured differences between the observational units. Simultaneous has the usual temporal meaning, but also carries a certain spatial component. Clearly, two patches of grassland on different continents are unlikely to serve as reasonable replicates of each other - there are simply too many things changing. However, two grassland patches in the same ecoregion may well work for comparing different burning practices.

In AM, the idea of an experiment is to use the management action itself to create the experimental treatments, and in that case the desire to determine causation beyond the current set of objects (e.g. management sites) is less important than figuring out which management actions work the best. An experiment will figure that distinction out quicker than applying treatments sequentially to a single object, because the simultaneity of the treatments helps to reduce the number of alternative explanations.

While it is true that society often conducts large scale manipulations of ecosystems without simultaneous alternative treatments, I do not believe it is helpful to describe these manipulations as experiments. If we do, then everything is an experiment, and the word ceases to have any value, much like the word sustainability or indeed, adaptive management. An experiment with simultaneous treatments is not the only way to distinguish between competing hypotheses, but it is a very good way when it is possible.