Experiment and replication

British Science week falls between 10 and 19 March 2023. This year the theme is connections. In the spirit of this theme, I offer some historical reflections on experiment and replication, looking to the connection between past and present practice. Psychology, along with Medicine, has been at the heart of the replication crisis over the last decade or more. My intention is not to rehearse the details of that crisis but rather to inspect the concept of experimentation and the role of replication.

Whilst experiments are to be found throughout history, as a core scientific practice it is most closely associated with developments in the 17th century [1]. This is the early Modern Science period, and more often than not the rise of experimental philosophy is seen as a specifically English practice [2], although some would disagree [1,3]. The reason for this English bias is the prolific work of Robert Boyle. Boyle both developed experiments and laid the foundations for reporting experimental work [4,5]. He was also instrumental in founding the Royal Society.

For Boyle the central role of an experiment was to produce a matter of fact. What this meant was to reveal a phenomenon, to assert it as a fact of nature in want of an explanation. This activity was regarded as the primary action for a natural philosopher and marked a distinct transition from the prior practice of relying on arguments from authority. More precisely, Boyle sought to remove natural philosophy from a priori reasoning founded in metaphysical principles, and instead to attach theorizing to matters of fact. This does not mean that Boyle had no metaphysical commitments – he was a corpuscular theorist – but rather that he sought to reduce their influence. In this way Boyle prefigured the ambitions of the logical positivists of the early 20th century who sought to ground science in observation statements.

Shapin argues that matters of fact were probabilistic statements – they were neither opinion nor certainty, both hallmarks of prior natural philosophy. This was a radical development in scholarship, but Shapin notes the industry of Boyle in establishing new scientific practices. Specifically, he claims that Boyle developed three technologies. First was the material technology. Famously in Boyle’s case this was the development of sophisticated air-pump apparatus. Whilst air pumps had existed prior to Boyle, along with Robert Hooke and others he improved their quality and shifted from metal to glass to inspect effects within the pump.

The second technology was a literary one. Boyle set up expectations for how to report experiments, giving clear and detailed accounts of all that was done without theoretical concepts informing the prose. He also made clear that both successful and unsuccessful experiments should be reported. The third technology, not unrelated to the literary, was a social one, establishing the rules of the game for natural philosophers when inspecting the epistemic claims made from demonstrated matters of fact.

Central to Boyle’s view on science, and to his literary and social technologies, was the concept of witnessing. In common with other thinkers, Boyle drew an explicit legal analogy [6]. The more corroborating witnesses there were, the more readily the community can assent to a matter of fact.  To achieve this, experiments were performed in public spaces, such as the Royal Society; reports of experiments listed credible witness present at the time of the experiment; and experimental reports were written up to enable their reproduction and with hope the replication of the phenomenon. This last was another way to multiply witnesses, but in most cases, it failed due to the material technologies deployed at the time. The manufacture of air pumps, for example, was complex and expensive. Moreover, it was difficult to impose a standard of manufacture and so air pumps necessarily differed, introducing novel errors to procedures and so forth.

In a recent paper about the replication crisis Boyle’s efforts are directly cited:

He cautioned against trying to understand the world by adhering to theoretical frameworks (‘systems’). Instead, he commends starting an investigation by conducting ‘a general survey of the subject’ before moving to meticulously designed and reported experiments.

He chastises those who fail to report experiments yielding results that are inconsistent with current theories. He stresses that the results of experiments should be seen as ‘provisional’: he repeats experiments himself and encourages others to repeat them as well. If these replications fail to confirm the results of previous studies, he urges consideration of why this may have been so. And all of this guidance reinforces the need for science to be seen as a collective endeavour. (4: 82)

Whilst it is not explicitly said, Bishop and Gill are clearly emphasizing the matter of fact focus of Boyle, what we might term scientific phenomenology – the establishment and study of phenomena. Moreover, in the context of their paper they are strongly endorsing this as a core aspect of modern science.

One interpretation of scientific phenomenology has been that individual experiments, and other empirical practices, collect idiosyncratic data. It is idiosyncratic for multiple reasons, including differences in material technologies deployed, populations used, historical periods sampled, etc. But, in aggregate these studies enable scientists to assert a phenomenon which can then become an object for theoretical scrutiny [7]. Here we see a demarcation between data, phenomena, and theory, but also a proposed relationship between them.

Turning to replication, Haig notes two kinds of justification in science. First, a belief is justified if it is generated through sound processes. This is the reliabilist justification. Second, theoretical accounts must adhere to the epistemic virtues, including that of coherence. Haig claims that it is the reliabilist justification that is at the heart of the replication crisis, which draws the focus toward the establishment of phenomena, rather than theoretical accounts of nature. 

Whilst Haig is clearly in favour of reliable findings, his concern is with accounts of science that claim replication as a gold standard. For Haig this is too strong, drawing attention away from the epistemic virtues involved in theory development and the role of predictive utility. He cites occasions where theories are favoured, not due to replication but coherence, for example Darwin’s theory of evolution by natural selection fitted the available data better than alternative ideas. This was not an outcome produced by replication, and it is hard to claim that this theory was not useful. Furthermore, he notes that replication is not the only route to reliability, discussing methodological triangulation at some length. And he notes that replication is not always possible in some scientific endeavours, such as the analysis of population level datasets, that simply cannot be recreated. Haig’s broader point, however, is that we should see both reliabilist efforts to detect phenomena, and coherent theoretical development from those phenomena as partners in science.

Boyle’s recommendation is to seek direct replication, to multiply the witnesses before moving to theory. Bishop and Gill, by strongly leaning on Boyle, appear to be endorsing this and Haig detects a general emphasis upon direct replication as a gold standard for psychology, as if direct replication is all that is required to generate theory. Given that Psychology reports its findings as statistical effects Haig questions the ability of direct replication of focal studies to determine phenomena. What Haig has in mind, as phenomena, are a different order of effect such as that labelled the Stroop Effect or the Flynn Effect. The former denotes a consistently derivable error in name retrieval caused by stimulus interference. The latter is a population level change in performance across time. Both have been asserted through multiple studies, and this pluralism of approaches is a form of conceptual replication, that enables phenomena to be established. This does not mean that we should not expect each study to be reliable. It means that a reliable study is not sufficient to derive phenomena, or theory. Such conceptual replications rely upon a process of abstraction, which is a modelling approach, and an aspect of theory building.

Those advocating for direct replication of studies as a gold standard for Psychological Science run the risk of relegating, or perhaps even removing, much of the epistemic work required to deliver a scientific account. Boyle was integral to the development of science in the early modern period, and his three technologies were remarkable achievements that enabled a proliferation of activity. Reliability of testimony was, and still is of utmost importance. Boyle was in the business of trying to assert phenomena, in an almost all or nothing manner. And his apparatus was a way of modelling a small portion of the natural world to do that. But reliability was in the service of greater goods, that in turn depend upon other epistemic activities.

1. Steinle F. 2016 Stability and Replication of Experimental Results: A Historical Perspective. In Reproducibility: Principles, Problems, Practices, and Prospects (eds H Atmanspacher, S Maasen), pp. 39–63. Hoboken, NJ, USA: John Wiley & Sons, Inc. (doi:10.1002/9781118865064.ch3)

2. Dear P. 1990 Miracles, Experiments, and the Ordinary Course of Nature. Isis 81, 663–683.

3. Vanzo A. 2016 Experiment and speculation in seventeenth-century Italy: The case of Geminiano Montanari. Stud. Hist. Philos. Sci. Part A 56, 52–61. (doi:10.1016/j.shpsa.2015.11.001)

4. Bishop D, Gill E. 2020 Robert Boyle on the importance of reporting and replicating experiments. J. R. Soc. Med. 113, 79–83. (doi:10.1177/0141076820902625)

5. Shapin S. 1984 Pump and Circumstance: Robert Boyle’s Literary Technology. Soc. Stud. Sci. 14, 481–520. (doi:10.1177/030631284014004001)

6. Sargent R-M. 1989 Scientific experiment and legal expertise: The way of experience in seventeenth-century england. Stud. Hist. Philos. Sci. Part A 20, 19–45. (doi:10.1016/0039-3681(89)90032-0)

7. Haig BD. 2022 Understanding Replication in a Way That Is True to Science. Rev. Gen. Psychol. 26, 224–240. (doi:10.1177/10892680211046514)