Friday, February 7, 2014

Back to the Future


One of my last actions as Editor-in-Chief of Acta Psychologica was to accept a manuscript for publication that is very timely given the current “crisis of confidence” in psychology. One of the paper's key points is that it is crucial to distinguish between hypothesis testing and exploratory research.

The paper chimes in with many critics of current practices in psychology when it asserts that [i]t is essential that these hypotheses have been precisely formulated and that the details of the testing procedure (which should be as objective as possible) have been registered in advance. Several journals, such as Cortex, as well as special issues of journals already require authors to preregister their submissions and the Open Science Framework offers an extremely user-friendly platform to do just this.

The paper makes a clear distinction between hypothesis-generating and hypothesis-testing research and argues that researchers regularly conflate the two, passing off exploratory research as confirmatory—a very apt description of current research practices. In exploring the data, the paper continues, researchers try to extract from the material what is in it but necessarily also what is accidentally in it. And thus the researcher proceeds by trying and selecting, whereby the selection is based on whether it promises to produce interesting (i.e., significant) results. By operating in this fashion the researcher is capitalizing on coincidences. The paper then goes on to explain what the problem with this practice is by using the example of rolling a die (hmm, where have we seen this example before?).

The paper ends with a rather stern conclusion: If the processing of empirically obtained material has in any way an “exploratory character”, i.e. if the attempt to let the material speak leads to ad hoc decisions in terms of processing, as described above, then this precludes the exact interpretability of possible outcomes of statistical tests. This conclusion resonates well with comments made by other researchers in the current debate. Perhaps it is not surprising then that the paper modestly acknowledges: this conclusion is not new.

Adriaan Dingemans de Groot (1914-2006)
So if the conclusion is not new, then what’s so special about this paper? Well, it was published 1956, the year that the Soviets invaded Hungary, Elvis Presley entered the music charts for the first time, and Dwight D. Eisenhower was re-elected as President of the United States. The current one, Barack Obama, hadn’t been born yet, John Lennon and Paul McCartney had not even met, and Donald Trump (probably) still had normal hair. The internet was decades into the future and so was the “crisis of confidence” in psychology.

The article that I'm talking about here was written by Adriaan de Groot, a Dutch psychologist, who became internationally famous for his research on thought in chess, which had a major influence on Nobel Prize winner Herb Simon (as well as on my former colleagues Anders Ericsson and Neil Charness). De Groot also developed an intelligence test that all Dutch children are required to take at the end of elementary school and which selects them for various tracks of higher education (I vividly remember taking that test when I was eleven).

A group of researchers from the University of Amsterdam, led by Eric-Jan Wagenmakers, has now provided an annotated English translation of De Groot’s article. This is fitting because De Groot held a professorship in research methods in the Department of Psychology at the University of Amsterdam. As I said at the beginning of this post, this article is currently in press in Acta Psychologica, which is also apt given the Dutch origin of that journal, but the author version can be downloaded here legally and for free from Wagenmakers’ site (also appropriate, given the Dutch stereotypical thriftiness).

In their annotations Wagenmakers and his colleagues make the sad observation that De Groot’s original article has been cited only twice (!) to date. I expect that this translated version will receive the number citations that the original already deserved.

I thank E.J. Wagenmakers for comments on a previous version.

3 comments:

  1. I hope this paper will convince researchers that preregistration of analysis plans is nothing more than Methodology 101.
    E.J.

    ReplyDelete
    Replies
    1. Indeed. I'm trying to convince all my collaborators of this and OSF makes preregistration a breeze.

      Delete
    2. This blog is a very nice tribute to a psychologist who really deserves it: A.D. de Groot. 'Modern' psychology can still learn a lot from him, and in my opinion, his Methodology (1961) should still be obliged literature for psychology students in general and also for psychologists who haven't read it. One thing: De Groot did not develop an intelligence test for children, but you can consider him as the intellectual father of the Cito, after a visit in the late fifties to the Educational Testing Service in Princeton, during which De Groot developed a kind of blueprint for the Cito. This Dutch testing and assessment company developed the famous 'Cito-toets' (Cito-test) to which you are referring, which is more a scholastic aptitude test than an intelligence test (although you could argue how it is used nowadays).
      Vittorio Busato

      Delete