The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.
With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.”
I’m not surprised by this at all. The same thing happens with funding applications as well. Bigger names are more likely to be funded regardless of the merits of the study.
This reminds one of the PAL revues of a lotta climate science…especially Michael Mann and his “hockey shtick”..
I have often said that peer review, as used by pseudoscientists AKA those who seek to advance an agenda other than the truth, is little more than the logical fallacy of argumentum ad verecundiam with a heavy emphasis on the verecundiam. That schools and governments get taken by big name con artists simply verifies my contention.
Exactly. Peer review functions according to basic human psychology. You’ll get some reviewers with knowledge and above all, integrity. And then, you’ll get..the rest.
There’ll be actual ignorance based on the comfort of tenure where the academic has no incentive to keep up with research. And above all, there is the rejection of another perspective. There’ll be envy and anger and fear at the possibility that this new perspective might have merit.
In research grants, the old guard big names rely on government funding to maintain their staff and the universities insist on this type of ‘extra funding’ from the government. So, new researchers have little chance of funding because the money goes to the old guard and the big universities.
In Canada, research funding is particularly difficult because of the requirement that the reviewers are bilingual. This drastically reduces the pool of possible reviewers and as a result narrows the focus of their mindset. Grants tend to be given within this entrenched mindset and there is little chance of genuine innovation.
The awkward part comes from the fact that this article was published in 1982.
Not to say it’s not as bad now, just that these findings aren’t exactly germane anymore.
Buddy system in full effect. Always has been, too. Anyone recall the BS that Einstein had to put up with in his early days?
The problem with peer review is not that it’s a Old Boy network. The problem is they lie about it and pretend its objective and impartial, when it clearly is not.
learned men,now discuss the the madness of the crowds ,which currently reject AGW..
in the future ,they will discuss the madness of the crowd, for accepting AGW..,
Spellbound raises an interesting, but not necessarily compelling, point. How often is it appropriate to do an empirical study of the blindingly obvious?
I think it is time to replace peer review with a more democratic way of evaluating ideas and scholarship. I would guess that a system such as that available on reddit would not be too difficult to implement. Comments could easily bring up questions and concerns that peer review sometimes does. Peer review (and most of those involved in that process) are ossified and need to update their thinking.
Funny how 97% of scientists believe in AGW but 100% of the output of their oh so expensive models is wrong.
Now that is a consensus.
Phantom – “Anyone recall the BS that Einstein had to put up with in his early days?” Or his own fights against quantum mechanics?
“Peer review” is not meant to involve as much as people seem to expect. Actually, it is only meant to be a fairly cursory scan looking for truly glaring errors. It should spot “Thomas Jefferson never owned a firearm” but not necessarily “Jefferson made his own guns.”
Of course, defining “peer” is, well, a bit arbitrary. While “Climate Scientists” swallowed “There was neither a ‘Medieval Warm’ nor a ‘Little Ice Age'” meteorologists probably would not have – at least not unless/until politics forced it.
JohnA said: “”Peer review” is not meant to involve as much as people seem to expect. Actually, it is only meant to be a fairly cursory scan looking for truly glaring errors.”
At which stated goal it is an abject, crashing failure. The true purpose of “peer review” in this day and age is to impose the patronage system on young scientists and provide a minor revenue stream for established ones. As we see from the present article, if you have a Big Name as lead author on a paper then it will be published. If you don’t, it won’t.
Reviewers don’t read the papers. They read the author list and if they recognize a name they pass it.
How else to explain the medical literature? Half the papers I see would be laughed out of a first year statistics course.
Note the reference shown clearly in the link:”Behavioral and Brain Sciences / Volume 5 / Issue 02 / June 1982, pp 187-195″
This article was originally published in 1982, and released on the web in 2010. The point is valid, but the study is over 30 years old. It would be interesting to see a current re-run of the study, and if standards have improved or declined.
This study shows problems with psychology departments, a notoriously “soft science”. Does it apply to the hard sciences? I don’t know, but you shouldn’t just make that assumption.
Likewise, this study shows how the sciences are always questioning themselves which is a sign of health, not of weakness. Though… changing course, and noting one’s own errors is a weakness for conservatives. That’s why you never see this kind of questioning by religious types, save for the Catholic’s Jesuits, and look what happens to those guys every few centuries. #burnedatthestake
It doesn’t matter whose name appears. — Fred Bloggs
It doesn’t matter whose name appears. — Sexy Naked Girl
John, you’re just daft.
One very efficient way to postpublication review articles would be by creating a database with reviews that have already refuted claims, so that critics can postpublication review by posting links to those articles wherever the refuted nonsense arguments crop up (no need to write the refutation as a new comment each time). New refutations can constantly be added to the database, including refutations that expose flaws in previous refutations. This may be the way to replace expensive, slow and partly arbitrary prepublication review that refuses to touch theories just because where they happened to be initially published.