When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.
If it’s true that one gets what one pays for, let me point out that referees typically work for no payment. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly. The peer review process itself is full of serious flaws, yet is treated as if it’s the handmaiden of objective truth.
I knew that a long time ago when I was a young grad student. My supervisor back then published some papers on the topic I was investigating. Reading through them, which were mediocre at best, it became apparent to me that he could have combined them into a single, but excellent, publication.
Instead, he took his set of data, divided it in two and published a paper based on one of them. One of those sets was further divided, resulting in two more. The result was three papers for the price of one and there were enormous benefits from that. He could add three more entries on his publication list, which would make him look good in his performance review. As well, he could mention those three papers to whoever funded the work and, since it was unlikely that anyone on that end would read them, he would have a good chance of getting the funding renewed.
What he did wasn’t illegal or necessarily unethical, but the underlying wisdom was certainly subject to question.
I noticed it again during my Ph. D. thesis defence. I could tell who actually read it based on the questions that were asked and who posed them.
I had a certain image about the system when I started grad studies. By the time I finished, I was close to being completely disillusioned.
“yet is treated as if it’s the handmaiden of objective truth.”
And that’s a pity….because we all know that Wikipedia is the only thing that does that.
Another thing that became apparent to me was that dissent from the accepted canon and even originality are discouraged if not punished.
Part of the reason is that the system is heavily dependent upon compliance and groupthink. Having a differing opinion is tolerated, but marginally so.
This is because researchers rely on external funding. They have to prove to whoever’s going to be financing the work that they will provide value for the money. In other words, they have to prove that what they’re investigating will turn out to be true or will be confirmed. If that doesn’t happen, then there will be questions about the validity of the results, how they were obtained, and whatever ideas, theories, and assumptions that the research was based upon.
Setting out to confirm that A is correct and then finding out that it isn’t can be hazardous to one’s funding and, ultimately, one’s career.
I think you can see where this is going. So, if one gets money to investigate, say, “climate change” and it turns out that it’s not the case, then it’s game over.
Now it doesn’t mean that researchers will use underhanded means to maintain their funding, though, I’m sure, there are some who do. What it means, though, is that few people are willing to stick their necks out when it comes to their research, taking a more conservative approach as there’s a higher probability that the results will turn out to be as one would expect them to be.
Another thing that’s wrong with the system is the fact that universities are nothing but money-harvesting machines nowadays.
In order to get tenure, one has to bring in external funding. In order to keep that “job for life”, one has to keep shilling for cash.
Don’t think that if one had one’s own money that one doesn’t have to be a beggar for financing. The issue is control of that financing. Just because the research funds are in one’s name, it doesn’t mean that one has complete access to it. I’ve heard of university departments insisting that all external funding be pooled and the money doled out accordingly. Having one’s own cash simply isn’t playing fair, at least according to those rules.
With such imposed restrictions on how one might be allowed to use that money, it’s little wonder that university researchers aren’t about to make any bold strides in their work.
Yeah, it’s a messy system, quite unlike how it’s portrayed to the public in Hollywood movies and PBS documentaries.
Great site that Not A Lot of People Know About!
Bookmarked!
Thanks for the link!
Here’s a humorous twist on the subject:
http://phdcomics.com/comics/archive.php?comicid=1900
http://phdcomics.com/comics/archive.php?comicid=1901
And what is required for gov’t. to accept this 50-50 coin flip of truth as “science” ? A poorly educated (STEM) population is all. A huge pool of taxpayers who, gaping mouth, eat up everything shoved down their gullets by the government as “science” … 97% confirmed truth, as certified by peers Scientific “certainty” that you are killing mother Gaia faster than ever in the history of the rocks, and that only a substantial tithe placed in the UN collection plate will save the planet for the children you will never have – because the one-world gov’t. forbids reproduction as means to control the population. All you need is ignorance. Mass ignorance. Masses distracted by Kardashians, pucks, and futballs. Peer review can be sloppy ! Who will ever notice … and if everyone keeps getting a nice big paycheck … who cares, right ? Who cares if we spend all our limited resources chasing a problem that is wholly invented by some clever statistical mind-fukcing. Dumb-down the the population (while simultaneously fattening the wallets of teachers Unions). Drop all standards and methods of education to reward ignorance. Reward make-believe substitutions for reality as “creative” and worthy of top marks. Water-down the entire grading and testing of students so there is no distinction between intelligence and bullsheet. Then, peer review will be utterly unnecessary … except as “confirmation” of the state-sponsored narrative.
Do you really have to give your academic resume every time you post?
If you want people to think you’re smart then make smart comments. What you say about yourself won’t do it.
Do you really have to give your academic resume every time you post?
If you want people to think you’re smart then make smart comments. What you say about yourself won’t do it.
If you’ve read anything that I’ve written, you’ll have seen that I’ve been quite critical of the post-secondary system.
I made similar comments while I taught at a certain post-secondary institution. Doing so didn’t make me popular with administrators and colleagues.
The administrators didn’t like it because I criticized the golden hog trough system that the post-secondary educational system had become for them. Some colleagues didn’t like my comments for the same reason, partly because they waiting to have their turn at that trough.
Others were uncomfortable with what I said because I said out loud what they were not only thinking and believed themselves, but were afraid to say it.
One reason I quit my job was because I realized that the system is broken beyond repair. Attempts to fix it were futile, if not hazardous to one’s career prospects as there were a lot of people (see my earlier comments) who had a vested interest in maintaining the status quo.
One reason I quit my job was because I realized that the system is broken beyond repair.
I left academia for the same reasons.
I found working as a bartender in a bordello was more honest employment and nicer associates, providing a higher quality of service, with much better job benefits.
I found working as a bartender in a bordello was more honest employment and nicer associates, providing a higher quality of service, with much better job benefits.
I’m sure there are people who would think it’s more respectable, too.
“Do you really have to give your academic resume every time you post?
If you want people to think you’re smart then make smart comments. What you say about yourself won’t do it.”
What motivated you to make a dumb comment like that?
“What motivated you to make a dumb comment like that?” RE: Smithereens comment to B
I’m just guessing … maybe multiple daily repetitive comments? However, I can’t speak for Smithereens.
My university research days are decades behind me now, but my experience about publishing papers was that if one is publishing a paper that makes a small contribution and doesn’t deviate from orthodoxy in the particular branch of science one is dealing with, it gets published quickly. If, however, one is trying to publish something novel and which goes against orthodoxy, then one has a lot harder time in doing this.
The last research I was involved in (before I decided to get away from the time-wasting exercise of having to apply yearly for grants so I would get paid and got into medical school) was very bleeding edge for the time and required computing power which wasn’t cheap in the 1980’s, a crash course in the mathematical methods I needed for the project, lots of hardware hacking and endless nights of programming let alone doing the experiments which required me to get into meatware hacking mode. The system eventually worked and the reviewers nitpicked every detail of our papers and, fortunately we managed to acquire a mathematician co-author who satisfied the reviewers needs for mathematical rigor. Out of curiousity, I’ve self-googled the paper titles for the last 15 years or so and, just recently found an increasing number of quotes of our work from almost 30 years ago. Of course, now a desktop PC has considerably more computing power than the $100 K worth of minicomputers and array processors that were required to do the initial experiments. I still think it was more fun to build the hardware to network PDP-11’s and write all the code myself rather than writing a MatLab “program” to duplicate our work.
I’ve also reviewed papers and viewed it as my job to read the paper thoroughly, check the references which the authors have used and work through the math to see if it makes sense. I’d get handed electrophysiologic papers where my boss couldn’t understand the math and it was my job to review the paper. Of course I’d find errors and these were duly corrected before the paper saw print. I just wish my mathematical skills were as good now as back then but it seems medical algorithms overwrote to a large extent the mathematical areas of my brain.
So, if you’re looking to publish without much fuss, don’t come up with any novel ideas (Einstein would probably have a horrific time getting into print now) and ensure that ones findings are confirmatory of the theories of the current “experts” in the field. That explains by 50-80% of scientific papers are essentially useless as their primary purpose is to pad the authors CV.
When I got into medicine, I was horrified by what passes as “scientific research” in the medical field. I’ve had numerous arguments with physicians when I tell them medicine isn’t a science and is best considered to be a skilled trade that utilizes the same pedagogic techniques as a master potter teaching an apprentice. Medicine relies more and more on concensus positions and this is dangerous. There is no statistical correlation between 10 senior respected physicians agreeing that a particular treatment is the best thing to do and what is actually the best thing to do for a particular patient. What’s made things worse is that much of medical research is pseudoscience where inappropriate statistics are used and produce conclusions contrary to what an experienced clinician whose intuition has been trained by decades of practice would conclude.
The medical literature is rife with stupidity and nowhere is this more glaringly apparent than when it comes to firearms. Dr. Suter published an excellent paper “Guns in the medical literature: a failure of peer review”. The CMAJ has been pumping out anti-gun papers for decades which are of such poor quality, one wonders how they ever got published. The primary reason is that the peer reviewers were as equally ignorant of firearms as the author. The same holds in “climate science” where the implication of the earths climate as a constrained chaotic system hasn’t sunk in at all. (Doctors are equally bad dealing with chaotic systems and the fractal nature of human physiology on an overt level but skilled physicians have tuned their intuition to be aware of human non-linearities). “Climate science” is driven by political considerations and the “researchers” in this area have sought to remedy their massive mathematical ignorance with all the computing power government money can buy. Analyses I’ve seen of some of the climate models have pointed out such simple errors such as not accounting for roundoff and quantitization errors which have a very nasty habit of becoming more and more of a problem as one does numeric solution of systems of coupled differential equations on longer and longer simulate time spans. One of the first things that I check out when I code stuff like this, but then I’m very old school in paying attention to details like this. As long as a paper produces results which look like the most alarmist publications of “experts” in the field, who needs detailed peer review?
I’ve missed your comments loki. Have I just been inattentive, or have you been absent for a while?
So, if you’re looking to publish without much fuss, don’t come up with any novel ideas (Einstein would probably have a horrific time getting into print now) and ensure that ones findings are confirmatory of the theories of the current “experts” in the field. That explains by 50-80% of scientific papers are essentially useless as their primary purpose is to pad the authors CV.
For my Ph. D. research, I created a computer model of the system I was investigating, basing it on a little-known mathematical technique.
From what I gathered, when that method first appeared, it wasn’t generally accepted by the research community at large. It was considered to be too unconventional and it seemed that its creator had a hard time publishing the original paper. Eventually, it was accepted and printed by some obscure European journal.
The author wrote a number of related papers and many of them were similarly published. Eventually, one was accepted by a journal produced by a well-known technical organization.
As for the method itself, I found it to be quite effective for what I was working on. By the time I was doing my thesis research, it had gained some degree of acceptance as a legitimate approach.
Skepticism and need for a higher standard of proof are appropriate for new methods. Here peer review is helpfull — that is if the reviewers actually dig into the details of the paper and check that it’s free of errors and conclusions are supported by data. Much easier now that authors will post their raw data online; something I would have done in the 1980’s if there had been a means to post the absolutely massive data files which totalled perhaps 100 Mb (this was when having 64 Mb of removable disk storage on a system was considered impressive).
New methodology takes a long time to make it into the literature. Currently am playing around with analyzing daylong chunks of human heart rate data using fractal analysis methods such as Hurst exponents and detrended fluctuation analysis (DFA). Mandelbrot published a lot of the background material in the 1960’s but, despite this the medical profession seems to be singularly ignorant of the potential diagnostic potential of fractal analysis of physiologic time series. The most frustrating time I’ve had in finding full text references is that most of the papers are published in physics or mathematical journals which are not available in the set of journals I have online access to through the BC college of physicians library and hospital libraries. Physionet.org has large collections of physiologic data one can play with as well as the mathematical tools one needs but most of the work I’ve found has been on peoples blogs and informal publications.
There was a very interesting psychology paper that I ran into (and can’t find link which is lost somewhere in my Tb of downloaded material) which looked at the dangers of homogeneity in the perversion of science. Medicine is a good example because of the emphasis on “consensus positions” and, when everyone thinks the same, the mistake of missing the elephant in the middle of the room is quite common. When one has a very disparate group of individuals, there will inevitably be one person who seems something that everyone else has missed. In a functional diverse group of scientists, an objective analysis of the effects of the unexpected black swan will be carried out. A true scientist doesn’t become overly attached to their theories and unexpected disproof is evidence that one has made a mistake somewhere along the line. Medicine is more akin to climate “science” in that “public health” is used as an excuse to forcibly impose political views of physicians involved in the area on populations “for their own good”. Much of the junk science on firearms in medical literature is as a result of urban doctors who’ve never touched a gun in their lives, and as they have no redneck colleagues, they’re ready to ban what they percieve as only dangerous.
The singular nature of the 2009 H1N1 influenza epidemic and the horrendous reactions I saw to the influenza vaccine that year, made me look at the whole issue of vaccination again and I was horrified by what I saw. No vaccine is subject to placebo-controlled double blind trials as the medical establishment thinks that to do so would be “unethical”. They are so convinced of the ubiquitous beneficial properties of vaccines that the “unethical” portion comes from denying the control group the “benefits” of vaccination.
Looking at infection rates and historical data from epidemics reveals a very non-linear process for which the mathematical tools used by physicans are totally inadequate; introducing a vaccine for an illness when the incidence is in rapid decline and claiming a “cure”. Like any natural highly non-linear and often chaotic process, interaction of groups of humans with infectious organisms can’t be modelled by simplistic Gaussian statistical processes.
Fortunately, online publishing has been providing an alternative to the ossified and increasingly group thinking “peer reviewed” journals and I now find most of my usefull information on peoples web sites or blogs. For example, bearcave.com, which is primarily a financial data analysis site, gave me the best explanation of how to compute Hurst exponents whereas the paper published in J. Mathematical Education was wrong. Most of the usefull materials I find have a much broader base of peer reviewers (if they allow comments) and crowdsourced peer review is, IMHO, superior to the process most print journals use. Publishing online doesn’t help to pad ones CV, but as I no longer have any intention of doing research in an academic environment, I don’t really care. The standards of site like wattsupwiththat.com and related climate sites are orders of magnitude higher than and climate “science” journal and the scientific expertise to provide croudsourced review of articles is staggering on those sites.
Skepticism and need for a higher standard of proof are appropriate for new methods. Here peer review is helpfull — that is if the reviewers actually dig into the details of the paper and check that it’s free of errors and conclusions are supported by data.
Genuine skepticism is welcome.
I remember when, about 20 years ago, possible evidence of bacterial life was seen in a meteorite found in Antarctica. The rock was determined to have originated from Mars, so the announcement of that evidence gained a great deal of public attention.
The researchers in question presented their findings at a press conference at about the same time as their paper appeared in Science. What I liked about this was that they asked other investigators to look at their results and were willing to be proven wrong.
I later read through that paper and, although what was presented was compelling, there wasn’t enough there, in my mind, that was persuasive.
Another problem that plagues the research community is groupthink. One example of that is nuclear fusion.
Right now, there are several ways of going about it. The one that has received the greatest amount of attention and, consequently, the most funding is the Tokamak. Several attempts to get one to function in an economically feasible manner, the latest being the ITER. It’s expensive, behind schedule, and whether it’ll work properly and as hoped is subject to question.
Meanwhile, there are researchers pursuing other approaches, few of which received as much publicity as the Tokamak, if any. One is inertial electrostatic confinement, sometimes referred to as the polywell. One group that investigated it built several test reactors and demonstrated the feasibility of the concept. Unfortunately, that work came to an end due to lack of funding.
So there’s a method which appears to work, but will probably gather dust because most researchers have jumped on the Tokamak bandwagon. The polywell, by comparison, had only a few people working on it with a considerably smaller budget.