John Dennehy has posted an interesting summary on The Evilutionary Biologist about professional peer review1. He notes, along with Marc Hauser and Ernst Fehr, that delays imposed by slow reviewers can be a significant source of frustration with the peer review process. The suggestion by Hauser and Fehr (2007) is to institute a system of punishments and rewards to get reviewers to submit reviews on schedule. Interesting idea, though I strongly oppose intentionally subjecting anyone’s work to delay as punishment, no matter how dawdling they are as reviewers. The scientific community at large should not be held back in order to punish specific individuals. There is also the obvious difficulty that reviewers may begin to substitute speed for quality in their review of manuscripts. I am currently reviewing four papers for four different journals. It will take time to get through them, and I hope to get them all in on time, but rushing them to meet a deadline won’t help the peer review process.
Long turnaround times are a real issue, and I have my own stories (one paper took over a year to show up in print). But my complaint regarding peer review comes as a reviewer rather than as an author. One of the biggest frustrations comes when one reviews a paper carefully, provides detailed comments, points out significant problems with the data, analysis, or interpretation, and recommends that the paper be rejected in its present format — and then it shows up in one’s mailbox again, unaltered, after simply having been submitted to a different journal, or worse, appears in print in another journal with none of the errors corrected. If anything shakes my confidence in the efficacy of peer review, it is this.
I understand full well the pressure to publish, but something has to be done about the tendency to submit a rejected paper — sometimes without even fixing typos that have been pointed out — to journal after journal (my current record is reviewing the same paper three times for three journals) until it gets through reviewers who are willing to let the mistakes slide or who lack the expertise to recognize the problems.
My suggested solution is that authors should be required to submit all previous reviews to any new journal to which they are sending the same paper. They should be required to show the editor that changes have been made or to justify why they have not. Otherwise, the peer review process is undermined, the quality of the science suffers, and the reviewers’ time is completely wasted.
End rant.
_________
Notes
1Not to be confused with the spectacle currently going on with regard to the flagellum paper.
References
Hauser M, Fehr E (2007) An incentive solution to the peer review problem. PLoS Biology 5: e107.
I am a fresh PhD and while I have gotten a couple of papers rejected, I have tried to take into account all the comments from the referees. The problem is that if I rewrite the paper according to one referee’s suggestions and then submit it to another journal, they seem to come up with entirely different comments, often contradictory to the edits I have made.
Perhaps it is only my limited experience, but I can see how playing a few rounds of this roulette might lead to frustration and deciding to resubmit with minimal changes. So actually as an author I welcome your suggestion too 🙂
Of course, sometimes reviewers don’t know what they are talking about, or simply hold a different opinion, or make unreasonable demands. You’re absolutely correct about that. But if you check the end of my post, my recommendation is that “[authors] should be required to show the editor that changes have been made or to justify why they have not“, which is exactly what you would have to do if the paper were not rejected but required major revision with the original journal. If anything, the third or fourth set of reviewers is likely to have less expertise than the first set that was contacted.
The problem as I have encountered it repeatedly is that some authors of rejected papers don’t fix anything at all, not even typographical errors or missing references, let alone demonstrably inaccurate statements.
Everyone has papers rejected, it’s a part of the process. Sometimes a paper is good, but it isn’t appropriate for the journal, or you get bad reviewers, or you aimed too high (Nature and Science reject a huge percentage outright).
The point is that there should be a requirement to submit previous reviews to new journals and to show that changes have been included or to rebut the reviewers’ comments. Just because you send to a second journal should not invalidate all the time and thought that went into the original review.
Plus, most fields have a relatively small number of experts in them, so it’s likely you will be getting the same reviewer in many cases. I have actually had to include a statement in a review such as “I have now reviewed this paper three times — do NOT send it anywhere else until you fix these errors”.
My suggestion: don’t let rejection get you down, take reviews to heart, but don’t be afraid to rebut a reviewer comment if you really feel you are not in error and can make a case.
The point is that there should be a requirement to submit previous reviews to new journals and to show that changes have been included or to rebut the reviewers’ comments.
No worries, we are in agreement 🙂
My point was that keeping the paper trail intact would not only help referees catch lazy authors, but may also help authors in case the paper is resubmitted to a different journal.
Although I can imagine some issues if the referees can read the previous reviews before they read the actual paper?
Although I can imagine some issues if the referees can read the previous reviews before they read the actual paper?
I am suggesting that these be provided to the editor, not to the reviewers. Reviewers do their reviews independently (although sometimes they will receive copies of the comments from the rest of the reviewers once a decision has been made).
To clarify, if you have a paper reviewed and the decision comes back as “resubmit after major revision”, you will have to show the editor what changes you made (or why you didn’t make them) before the paper will be accepted by that journal. If the paper is rejected and you have to submit it somewhere else, you should not get a free pass on the original reviews. What I have learned is that recommending rejection of a bad paper can actually make it more likely to appear in print than recommending major revisions but not rejection. That seems backwards, and it’s a symptom of the problem of authors just trying one journal after another with no effort to address criticisms if a paper gets rejected. Yes, there is pressure for publication — but we must be scholars about it.
Heh, so last night I just finished Ridley’s “The Cooperative Gene” and reached for the next book from the stack on my bedside table, and it was….
The Evolution of the Genome
Huh, I thought, T.R. Gregory, where have I seen that name before?
Anyway, encouraging timely reviews is a thorny problem. Perhaps in lieu of outright punishments, public databases could be created showing those who consistently produce late reviews. Public shame might encourage speedier reviews.
As for your complaints as a reviewer, getting unaltered resubmissions is disheartening. On the other hand, you already have a negative review prepared. If they recycle, why shouldn’t you?
As for your complaints as a reviewer, getting unaltered resubmissions is disheartening. On the other hand, you already have a negative review prepared. If they recycle, why shouldn’t you?
Oh, I do recycle them, but I still have to go through the two drafts to see if they have changed anything. It’s a far bigger problem when they show up in print in another journal when I did not see it the second time, and the editor was not privy to my initial review (and that of the other reviewer — it takes at least two to reject usually). It seems like a very small step to be required to address previous reviews, which is what one has to do anyhow if the paper were *not* rejected but require revisions.
I hope you enjoy the book. 🙂
“It’s a far bigger problem when they show up in print in another journal when I did not see it the second time.”
I agree. But how would you ensure an author submits previous reviews? Honor system? The devil is in the details. Another way to remedy this is thru post-publication “peer review” as Nick Matzke did with Liu and Ochman. Perhaps journals could make it easier to comment on published work like PLoS ONE has.
But how would you ensure an author submits previous reviews? Honor system?
We rely on the honour system in all other aspects of peer review, so I don’t see why it coulnd’t be applied here as well. To be more official, authors could sign a statement that they have provided all previous reviews. If a paper showed up in print that I had reviewed but rejected, I could ask the editor for confirmation that he saw my review.
Another way to remedy this is thru post-publication “peer review”…
Perhaps, but that happens anyway over time as the report is evaluated by the broader community.
… as Nick Matzke did with Liu and Ochman.
In my opinion, this bears little relationship to actual peer review and is far more harmful than helpful at this point.
Just curious if you wrote a letter to the editor about the papers that finally found a home after multiple rejections with no changes? That might make a difference slowly but surely.
I have yet to do anything proactive about it. I suppose I should someday…