It’s the building, stupid, not the builders!

Bands like The Beatles, the Jimi Hendrix Experience, Led Zeppelin, and Queen have spawned many tribute bands that produce note-for-note faithful renditions of original tunes like Can't Buy Me Love, Hey Joe, Stairway to Heaven, and Bohemian Rhapsody. They even look the part, with wigs, generous amounts of exposed chest hair, spandex, and vintage guitars. No one will applaud these tribute bands for their creativity but a certain segment of fans will appreciate being given the opportunity to relive the 1960s or 1970s.

In psychological science, replications of original studies are rare—at least in the published literature. And this is not the only difference with tribute bands. Another difference is that replication attempts are hardly viewed as tributes. No one is going to redo someone else’s experiment just for sentimental reasons (there is no scientific equivalent of a J.R. Stroop or Hermann Ebbinghaus tribute band). So why do replication studies?

Replication studies should be carried out to check the structural integrity of the edifice we call scientific knowledge. The goal is not to point fingers at the mason or apprentice who used a cracked stone or forgot to put in the necessary reinforcement (I don’t know the first thing about masonry, so I hope these examples make sense to all the building experts who are no doubt reading this blog). The goal is to erect a stronger building. Just as structural integrity checks are about the building and not the builders, replication attempts should be about the facts and not the researchers. Except for some shady contractors who are trying to let veneer walls pass for structural ones (recent examples happen to be social psychologists whose last names start with an S), we’re all working together on this building of ours.

This appears to be the spirit in which the Reproducibility Project is operating. Its members are trying to assess how reproducible the results are that three prominent psychology journals published in 2008. I am very interested in seeing the results. Not because they will allow us to tell which journal “won” (the smart money is on the Journal of Experimental Psychology: Learning, Memory, and Cognition) but because, taken together, these results will point to ways in which we can do our science better: replace some cracked stones, reinforce some sections, make a few additions, and sometimes perhaps abandon work on an entire wing. Moreover, they might lead the way to developing better building techniques.

My approach to replication is simple. I am not going after a series of results I find hard to fathom (otherwise, I’d be trying to replicate studies showing that priming with “professor” leads to better performance on a general knowledge test) or research by people I don’t like (I am fond of everyone). I am also not going to try to replicate a random sample of studies some of which might be on topics I don’t really care about that much (come to think about it, there are more than I would like to admit). My approach is thematically oriented.

I am interested in language comprehension and specifically in the mental representations that are involved. Are they word-like or more perception-like? I want to know how firm the foundation is of the little wing (subtle Jimi Hendrix reference) in that my fellow researchers and I have been laboring at. I want to know this so that we can develop better theories and methods (and not to kneecap a fellow researcher).

Together with Diane Pecher I recently published a series of replication attempts along these lines. It represents a first attempt. In an upcoming blog, I will talk more about the how and why of this research. The point here is that we should be concerned about the building but not the builders, except for the very small minority of them who are charlatans (more about them in a later blog as well).