New England Journal of Medicine;
Since it was first reported 3 years ago, when South Korean researcher Hyung-in Moon admitted to having invented e-mail addresses so that he could provide “peer reviews” of his own manuscripts, more than 250 articles have been retracted because of fake reviews — about 15% of the total number of retractions.
How is it possible to fake peer review? Moon, who studies medicinal plants, had set up a simple procedure. He gave journals recommendations for peer reviewers for his manuscripts, providing them with names and e-mail addresses. But these addresses were ones he created, so the requests to review went directly to him or his colleagues. Not surprisingly, the editor would be sent favorable reviews — sometimes within hours after the reviewing requests had been sent out. The fallout from Moon’s confession: 28 articles in various journals published by Informa were retracted, and one editor resigned.
What? No suicides …. it would be the decent thing to do.
Sure glad shit like this ain’t happening with “climate science”…..
Oh yeah?
Before the interweb, you had to work a lot harder. I know of a case in England where an ex-convict got through positive vetting system of an armoured car company by using the post office’s mail redirection service. Letters of enquiry to his former “employers” were forwarded to his associates in prison for glowing references, he was hired and after he had been working for a couple of years they got out and joined him for the robbery.
Bingo!
“The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America’s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.”
http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble/
Maybe they should study the hypothesis that the amount of government funding going into an area of science is proportional to the amount of unreproducibility.
I’m guessing a rough correlation of around 97%. 🙂
Would it surprise anyone that the inability to reproduce studies are worse the more subjective the branch of science? The humanities and social science researchers have an easier time fooling themselves and others.
It starts with the sample subjects who are not at all representative of the population. Does anyone who’s thought about really believe that a 20yo university student thinks and reacts to relationships, economics, sharing, value of higher ed, trust in institutions etc. the same as a middle aged, married parent? So even if you could reproduce the study with young students it would still not be valid. When the initial data is skewed then the results are meaningless.
Like I’ve said before, scrap peer review and replace it with an adversarial review. That solves two problems : researchers will have to clean up their studies or risk looking like idiots, the academic heterodoxy that currently exists will be better balanced by this diversity of thought.
The problem is that no one wants to do reproducibility studies since there’s neither money nor glory. Maybe money from the initial grant should specify that funds for adversarial confirmation must be set aside. This might not be a perfect system but surely it would be better than the 25% or less validity that currently exists.
This isn’t an indictment of peer review — quite the opposite.
It demonstrates what happens when the peer review process breaks down.
Peer Review Is Not What You Think
Shannon Love says this nicely [edited]
=== ===
“Peer Review” says nothing about conclusions. It is the fate of most scientific papers to be proven completely wrong.
Peer review protects a journal’s reputation. The journal hires experts to check for basic errors in math or methodology, along with grammar and spelling. It offloads responsibility for publishing bad papers onto anonymous scientists. It is a form of blame-passing that everyone would like to use. It does not confirm or refute experimental or theoretical conclusions.
Some people will say that a scientific result is true because it appears in a peer reviewed journal. That is the weakest defense possible. It means only that some editor and his reviewers found it to meet their minimum quality standards for publishing. It meets no standards if the editors and peer reviewers are corrupt.
=== ===
When people see “peer review”, they usually think of “scientific review”, which is the detailed investigation of data and the replication of results by independent scientists. Scientific review gives some confidence that the claimed results are correct. Even then, conclusions about what the results mean can be totally wrong.
Many of the scientists contributing to the IPCC (especially at East Anglia University) were committed to stopping scientific review. Despicably, they used their power of peer review to exclude criticism of their papers, and they refused to release underlying data to any organized scientific review. Fortunately, a few scientific bloggers were able to make many of these faults public.
Amazingly, the IPCC didn’t even restrict itself to peer reviewed results. The IPCC is a political institution, not a scientific one. Being half a scientist is like being half a truth.
One reform would be to expect that journals list the papers that they reject for publication, as a matter of scientific integrity. This would hold the journals reponsible for their editorial opinion and their assumed gatekeeper role. They would be more careful and show less bias, with the journal’s reputation on the line.
It would be fine to go further and give the reviewer’s opinions. This would encourage publishing pro and con views for rejected papers, and possibly also for accepted ones.
The great, late physicist Richard Feynman fought for open, transparent science. He wrote:
=== ===
“When someone says, ‘Science teaches such and such,’ he is using the word incorrectly. Science doesn’t teach anything; experience teaches it. If he says to you, ‘Science has shown such and such,’ you should ask, ‘How does science show it?’
How did the scientists find out? How? What? Where?’ It should not be ‘science has shown.’ And, you have as much right as anyone else, upon hearing about the experiments and after hearing all the evidence, to judge whether a sensible conclusion has been arrived at.”
=== ===
( easyopinions.blogspot.com/2009/11/peer-review-is-not-what-you-think.html )
Peer Review Is Not What You Think
Shannon Love says this nicely [edited]
=== ===
“Peer Review” says nothing about conclusions. It is the fate of most scientific papers to be proven completely wrong.
Peer review protects a journal’s reputation. The journal hires experts to check for basic errors in math or methodology, along with grammar and spelling. It offloads responsibility for publishing bad papers onto anonymous scientists. It is a form of blame-passing that everyone would like to use. It does not confirm or refute experimental or theoretical conclusions.
Some people will say that a scientific result is true because it appears in a peer reviewed journal. That is the weakest defense possible. It means only that some editor and his reviewers found it to meet their minimum quality standards for publishing. It meets no standards if the editors and peer reviewers are corrupt.
=== ===
When people see “peer review”, they usually think of “scientific review”, which is the detailed investigation of data and the replication of results by independent scientists. Scientific review gives some confidence that the claimed results are correct. Even then, conclusions about what the results mean can be totally wrong.
Many of the scientists contributing to the IPCC (especially at East Anglia University) were committed to stopping scientific review. Despicably, they used their power of peer review to exclude criticism of their papers, and they refused to release underlying data to any organized scientific review. Fortunately, a few scientific bloggers were able to make many of these faults public.
Amazingly, the IPCC didn’t even restrict itself to peer reviewed results. The IPCC is a political institution, not a scientific one. Being half a scientist is like being half a truth.
One reform would be to expect that journals list the papers that they reject for publication, as a matter of scientific integrity. This would hold the journals reponsible for their editorial opinion and their assumed gatekeeper role. They would be more careful and show less bias, with the journal’s reputation on the line.
It would be fine to go further and give the reviewer’s opinions. This would encourage publishing pro and con views for rejected papers, and possibly also for accepted ones.
The great, late physicist Richard Feynman fought for open, transparent science.
( easyopinions.blogspot.com/#trueScience )
He wrote:
=== ===
“When someone says, ‘Science teaches such and such,’ he is using the word incorrectly. Science doesn’t teach anything; experience teaches it. If he says to you, ‘Science has shown such and such,’ you should ask, ‘How does science show it?’
How did the scientists find out? How? What? Where?’ It should not be ‘science has shown.’ And, you have as much right as anyone else, upon hearing about the experiments and after hearing all the evidence, to judge whether a sensible conclusion has been arrived at.”
=== ===