It's a frustrating (and frustratingly common) experience for the author. But it also contributes to the systemic overburdening of journals, as the author now needs to (perhaps make minor tweaks and then) send their paper out to a new journal, which must find all new referees. It would've been more efficient if the original journal could have disregarded the confused report, and just sought the one replacement referee.
How could the journal know that the report was confused? Well, suppose that before rendering their verdict on your paper, the editors invited you to (briefly!) address the referee reports, indicating any major points of disagreement, and (roughly) what changes you'd make to it if given the opportunity for revisions. The editors would then make their final decision in light of both the referee reports and the author response.
This would require substantive philosophical judgment on the part of the editors, to adjudicate the inevitable disagreements between authors and referees (as does the current system, one would hope, when referees recommend diverging verdicts). Often they'd opt to reject anyway, no doubt. But other times the author's case might be sufficiently clear-cut that the editors are able and willing to disregard a confused referee report. And that could be a good thing all around, if the new referee recommends acceptance: better for the author, certainly, but also better for the profession as a whole (by reducing the burden on journals generally), and for this particular journal (that might now publish an excellent paper that they would otherwise have mistakenly rejected).
Of course, a bad author response might tip the journal towards rejecting what they would otherwise have R&R-ed, if they judge that the author's line of response is inadequate to the task. That too is a good result, since it cuts down on the time wasted on an ultimately negative verdict.
Relatedly: there would be little point in eliciting author responses if the original reports were uniformly negative: as with desk rejections, editors can helpfully save everyone time by rejecting a paper they judge to have little chance of being worth pursuing further. So my proposal should perhaps be limited to cases involving either split referee verdicts, or at least sufficient positive indications as to be worth the bother.
Would such a process be excessively onerous on editors? I'd welcome their thoughts. My (admittedly ignorant) guess is that it'd add an extra couple of minutes per submission to read the author's response alongside the referee reports. If enough journals started doing this, this time might well be counterbalanced by the overall reduction in submissions. So the gains from improving the system strike me as worth these up-front time costs. But I'm not a journal editor, so my opinion isn't one that really matters here.
Let me close by noting a way that these benefits might be realized without requiring any systemic changes to the way that journals are run. Referees might unilaterally choose to write their reports with a bit more humility (at least if the paper is sufficiently interesting to merit it), preferring a verdict of R&R to allow authors the chance to address their "devastating objection," even if they're doubtful of the prospects.
Sometimes this does happen, of course. (Just not often enough.) I've had good experiences of this sort on both sides of the author/referee divide. I've come to deeply appreciate referees who frame their reports in sufficiently constructive ways as to encourage revisions rather than outright rejection. And I try to do the same in return, when I referee an interesting paper, even if I personally think that it is deeply flawed. (There's no reason to be so charitable with uninteresting papers, of course. And, given widespread disagreements about what's philosophically interesting, this might well explain some of the problem here: even referees who agree with my norm here might, despite their superficially bad objections, actually be rejecting your paper because they find it comparatively uninteresting. In which case: fair enough, there's not much that can be done about that, except to hope for more sympathetic referees at the next journal. Or push for the systemic reforms above that would give papers with some positive indications -- e.g. from the other referee -- more of a shot.)
P.S. This proposed reform could be best combined with higher publication rates at top journals, most easily achievable if they switch to an online-only (and preferably open access) model -- which would also be good for independent reasons.
Posts
Some of my colleagues and I have joked about setting up a Court of Philosophical Appeals. Disgruntled authors can submit their papers, along with the referee reports... we review the case and decide whether to rule in favor of the author or Reviewer #2.
ReplyDeleteHa, very good!
DeleteA few things: (1) I'm not sure that this is true: "the top journals...tend to outright reject any paper that doesn't receive uniformly positive peer reviews." This is not true with respect to the two journals for which I've served as an Associate Editor. It may be that the default is to reject a submission with two reviewers who each recommend reject, but, even in such cases, there is usually a lag time that's in place so that the associate editor responsible for that submission can read the reports and intervene if they want to before a rejection is sent out by the managing editor. (2) You ask: "How could the journal know that the report was confused?" The answer is because the handling editor has read the submission themselves and can see that the objection raised is confused. Another possibility is that, after a rejection is sent out, the author sends a note to the handling editor anonymously through the managing editor explaining the referee's mistake. [Authors should do this very sparingly and only in the most egregious cases.] (3) Many authors, like yourself, seem to assume that editors reject a paper only on the basis of what the reviewers say in the comments that they pass on to the authors. This is not true. The handling editor reads the paper him- or herself. The handling editor also reads the comments that are for the editor's eyes only. Sometimes, the handling editor gets input from other associate editors. And sometimes the handling editor gets other feedback from the reviewers -- e.g., a numerical assessment of its likely importance. In many cases that I've been involved with, the handling editor has noted that a reviewer's objection is off-the-mark, but has decided on the basis of their own reading and on other feedback that they received that the paper just isn't likely to be important enough to warrant giving it a revise and resubmit. (4) Lastly, it's not a reviewer's job to outline every sufficient reason for rejecting a paper in the comments she passes on to the author. As a reviewer, what I put down in the comments for the author are those comments that are most constructive, helpful, tactful, and easily articulated. I don't list every reason or set of reasons that I take to be sufficient for rejecting a paper. So, editors have to give weight not only to the particular objections that the reviewer has passed on to the author, but also on their overall judgment and on any further less tactful comments that the reviewers passed on only to the editor.
ReplyDeleteThanks Doug, that's very helpful for perspective. (Your third point strikes me as especially important.)
DeleteLet me add one more thing. It seems to be a general misconception that when papers are rejected, the one to blame is the referee or referees who recommended rejection. But I think that it's ultimately the responsibility of the editors. We also read the submissions. We choose who to ask to referee. And we ultimately decide how to weigh all the evidence: the reports, our own assessment of the submission, our assessment of the reports, our assessment of the comments passed on only to us, and our assessment of the reliability of the testimony of the reviewers. So, if your paper is rejected and shouldn't have been, it's the editors who are to blame. Thus, editorial work is a huge responsibility. And we undoubtedly make mistakes. But, in my experience, editors are generally very conscientious and hardworking.
ReplyDeleteHi Richard, in my academic field (theoretical physics), this is common practice.
ReplyDeleteEven after a straight-up rejection (i.e. not a recommendation to R&R), authors are usually able to resubmit their paper to the same journal, with comments replying to the referee(s). Usually, the paper and the response are then sent back to the same referees (or in some cases, new ones) who write a new report and decide whether to modify their recommendation. Editor permitting, there might even be a 3rd round of review.
When there are multiple referees, they can see the previous reports of the other referees as well, and if they like comment on whether they agree with their criticisms.
Ultimately it is the editor's decision who to side with, but one can frequently appease the referees by making relatively minor changes to accommodate their points, or by clearly explaining why their objections do not apply.
Some journals, like Physical Review, also allow for a scientific-based appeal after all this, which goes to a member of an editorial board who can review the original editor's decision. Researchers rarely attempt this, since it's usually easier to just submit to some other journal, but I have successfully appealed one paper by this method.