Tuesday, December 8, 2015

Stepping in as Reviewers

Some years ago, when I served on the Academic Integrity Committee investigating allegations of fraud against Dirk Smeesters, it fell upon me to examine Word documents of some of his manuscripts (the few that were not “lost”). The “track changes” feature afforded me a glimpse of earlier versions of the manuscripts as well as of comments made by Smeesters and his co-authors. One thing that became immediately obvious was that while all authors had contributed to the introduction and discussion sections, Smeesters alone had pasted in the results sections. Sometimes, the results elicited comments from his co-authors: “Oh, I didn’t know we also collected these measures” to which Smeesters replied something like “Yeah, that’s what I routinely do.” Another comment I vividly remember is: “Wow, these results look even better than we expected. We’re smarter than we thought!” More than a little ironic in retrospect.

On the one hand I found these discoveries reassuring. I had spent many hours talking in person or via Skype with some of Smeesters’ co-authors. Their anguish was palpable and had given even me a few sleepless nights. The Word documents seemed to exonerate them. We had asked Smeesters to indicate for each study who had had access to the data, which he had dutifully done. For each study deemed problematic, he indicated he had sole access to the data and the Word documents confirmed this. I was relieved on behalf of the co-authors.

On the other hand, I found the co-authors’ lack of access to the data disturbing. You could fault them for apparently being uninterested in seeing the data and Smeesters for not sharing them. But how common it is to share data among co-authors anyway, I wondered? Smeesters obviously had his reasons for not sharing the data but there are also far more innocent reasons why co-authors may not want to share the data. For example, researchers may find it unpleasant to have somebody looking over their shoulder, as it might imply a perceived a lack of competence on their part. “I’m a Ph.D. now, I can analyze my own data, thank you very much.” Not trying to cause offense may make co-authors reluctant to ask for the data. Sometimes, the researcher analyzing the data may have used idiosyncratic steps in the process that are not easy to follow by others. Sharing the data would be onerous for such a person because this would require making every step that is second nature to them explicit for the benefit someone else. The perceived burdensomeness of the task could be another barrier against sharing data.

If there are barriers against sharing data among co-authors, then one might expect that the barriers against sharing data with third parties, such as reviewers, and other interested researchers are substantially higher. Indeed, this turns out to be the case in psychology, even after the turmoil that the field has recently gone through.

It seems that we like to play it close to the vest when our data are concerned. But science is not a poker game. When we take a few steps back from our own concerns, this becomes clear. We need to back up our claims with data and not with a poker face. We also have a responsibility towards our fellow researchers. Sure, they may be our competitors in some respects but together we’re in the business of knowledge acquisition. This process is greatly facilitated when there is open access to data. And finally, we have a responsibility to the society at large, which funds our research.

For these reasons, I’m proud to be part of the Peer Reviewers’ Openness Initiative. The basic idea behind the Initiative is that reviewers can step in to enhance the openness of our science. They do this by pledging not to offer comprehensive review for, nor recommend the publication of, any manuscript that does not meet several minimal requirements, which you can find on the website. I’ll just highlight three of them here.

(1) The data should be made publicly available.

We just discussed this.

(2) The stimuli should be made publicly available.

Just as we all benefit from access to data, we also benefit from access to stimulus materials. I cannot speak for other areas in psychology but in cognitive psychology, the sharing of stimuli has been common for decades. Back in the day it was not possible to have a printed journal article with a 10-page appendix with stimulus materials. Authors would provide a few sample stimuli and there would be a note that the complete materials were available from the corresponding author upon request. In my experience, the stimuli were always promptly sent when requested. Since the advent of the internet, there are no physical or financial limits to posting stimuli. At least for cognitive psychologists, therefore, this second PRO requirement should not be different from what is already common practice in the area.

(3) If there are reasons why data and/or stimuli cannot be shared, these should be specified.

It is important to note here that under the PRO Initiative, reviewers provide no evaluation of these reasons. In other words, under the PRO Initiative, reviewers are by no means arbiters of what counts as a valid reason and what not. The only requirement is that the reasons become part of the scientific record.

My father was a chain smoker for most of his life until he declared at one point:  “smoking is a filthy habit!” (“You can say that again!” I remember replying.) After this epiphany, my father never touched a cigarette again. I hope that the PRO Initiative will contribute to the field reaching a similar epiphany about lack of openness.  

If you’ve already had this epiphany, you may wish to sign the Initiative here.

For other views related to the Initiative, see blog posts by Richard Morey and Candice Morey.