Thursday, April 14, 2016

The Open Review Survey: First Peek

A few days ago, I conducted a survey among psychologists about open reviews. More than 250 colleagues volunteered their time, for which generosity I'd to thank them.

The survey was prompted by my being senior editor at Collabra, a journal that has the option of open reviews (other journals that I know have this feature are PeerJ and BMC Psychology). A fellow editor and I were wondering about people's views of open review. I conducted an initial quick survey on Twitter and followed this up with the more extensive one I'm describing here.

Participants were recruited via social media (Twitter, Facebook, LinkedIn) and the survey was open between Friday, April 8 and Tuesday April 12.

The survey consisted of open and closed questions. The former were included because I didn’t want to steer the participants too much; I was interested in their views. However, analyzing the open responses takes quite a bit of time, so here is an initial peek at the quantitative data only. I’m planning to write a paper that includes all of the data. (If editors are interested, please contact me.)

Although I specifically asked psychologists to participate, there were some respondents from other fields as well but their number is not large and some of the fields are likely close to psychology (e.g., communication, behavioral economics, psycholinguistics), so I’ve decided to include all respondents here.

My field of inquiry is

Answer
Response
%
psychology
250
90%
other social science
18
6%
natural sciences
9
3%
humanities
2
1%
Total
279
100%

The sample contains sizeable numbers from the ranks of graduate student to full professor; there are additional responses from a few undergraduate students and people with an “other” position; I’m assuming the latter are research associates. 

My current position is
Answer
Response
%
graduate student
66
24%
postdoc
54
19%
assistant professor
52
19%
associate professor
49
18%
full professor
40
14%
other
13
5%
Total
279
100%

I was interested in people’s views on open reviews both from the perspective of a reviewer and from the perspective of an author. Let’s first look at the reviewer perspective.

As a reviewer, I'd like to have my review published along with the accepted paper
Answer
Response
%
Agree, and I would like to sign my review.
108
40%
Agree, but only if I can remain anonymous.
87
32%
Disagree.
77
28%
Total
272
100%

Perhaps surprisingly, a large majority of respondents (72%) prefer to have their review published along with the article. For a third of the respondents, open reviews are desirable only if they can remain anonymous.

One concern that editors might have is that having an open-review option would decrease the likelihood that people will accept requests to review. However, as the results show, the prospect of open reviews would not make a difference for the majority of respondents. For about a quarter of the respondents, the likelihood would increase and for a fifth it would decrease. 

If there was an opportunity for open review, the likelihood that I'd accept an invitation to review would ….. if there were open reviews.
Answer
Response
%
increase
60
24%
decrease
51
20%
neither increase nor decrease
144
56%
Total
255
100%

How do open reviews look from the perspective of an author? Slightly more than half the respondents would like to have open reviews, while a third has no preference. Only 12% of the respondents would prefer not to have open reviews.

As an author,  I'd like to have open reviews
Answer
Response
%
Agree
137
55%
Disagree
29
12%
No preference
85
34%
Total
251
100%

Do response patterns vary as a function of having tenure? The perception exists that having (signed) open reviews, or maybe even unsigned ones, might make one vulnerable to retaliations. This would especially be worrisome for pre-tenure researchers.

To address this question, I collapsed across undergraduates, other, graduate students, postdocs, and assistant professors. I realize that there are considerable differences among these groups along many dimensions, but what they have in common is that they are pre-tenure. Why didn’t you directly ask about tenure?, you might wonder. I considered doing this but decided against it because it would make the issue of tenure more salient and might create demand effects.

Here are the, perhaps surprising, results.


Among the untenured respondents, a large majority (78%) favors open review. Of this latter group, half is even prepared to sign the reviews. Puzzling, to me at least, is the relatively large percentage of tenured respondents who are against having their review published (41%), almost twice the percentage found among untenured respondents! I'll look at the open responses in a later post to see what's going on here. Of the tenured respondents who favor open review, there appears to be less concern about being identified than among the untenured respondents. (Exercise: run your own Chi-square test here.)

Untenured respondents seem somewhat more likely to accept invitations when reviews are open than tenured respondents although openness wouldn't matter either way for the majority in each group.



How does open review look from the perspective of an author? Here, the pattern for the two groups is quite similar. 



So there you have it: our initial peek at the data. Of course, there are limitations to this survey. For one, it is not clear that this sample is representative for psychology at large. Perhaps it contains a larger percentage of proponents of openness than one would find in the population. Furthermore, there are other questions I could have asked. However, I wanted to keep the survey under five minutes to maximize participation.

With this in mind, it is possible to draw some conclusions. There clearly is support for open reviews, especially from prospective reviewers. However, the prospect of open reviews does not have much impact on researchers' willingness to accept an invitation to review. The support for open reviews is greater among untenured than among tenured researchers. However, for those in favor of open reviews, tenured faculty are less concerned about singing their reviews. Prospective authors are more in favor of open reviews than against and this does not differ between tenured and untenured researchers.

In the next post post I delve into the written responses.

8 comments:

  1. I think there is a mistake: the first and third table showing tenured vs. untenured appear to be the same.

    ReplyDelete
    Replies
    1. You're right. I fixed it. Thanks.

      Delete
    2. Thanks and great survey! It appears that one way to frame the results is to say that more experienced researchers are more skeptical of open reviews. Related to that, I wonder whether it makes sense to include grad students because they typically have little or no experience with peer-review. Looking forward to reading the paper with the full analysis.

      Delete
    3. Thanks! I agree that a more direct comparison would be postdoc/assistant prof vs. assoc/full prof. This would show pretty much exactly the same as with the students included.

      Delete
  2. My theory for the surprises....

    1) for untenured professors, there is more value in any publication, so makes sense to favor open.

    2) tenured profs, also might value the anonymouty for multiple practical reasons out of experience and relationships etc.

    ReplyDelete
    Replies
    1. But the tenured respondents seem split between those opposed to open review and those in favor signed open reviews.

      Delete
  3. Did you ask about gender? My intuition is that women would be less likely to favour signing their reviews but it would be interesting if that were wrong

    Agree that age/experience is a confound with tenure/non-tenure

    ReplyDelete
    Replies
    1. Interesting hypothesis. What would be your reason for entertaining it? Unfortunately, I didn't ask about gender.

      Delete