The fallibility of scientific journals | Publish and be wrong | The Economist
Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong. Now, along with Neal Young of the National Institutes of Health in Maryland and Omar Al-Ubaydli, an economist at George Mason University in Fairfax, Virginia, he suggests why.
PLoS Medicine – Why Current Publication Practices May Distort Science
The current system of publication in biomedical research provides a distorted view of the reality of scientific data that are generated in the laboratory and clinic. This system can be studied by applying principles from the field of economics. The “winner’s curse,” a more general statement of publication bias, suggests that the small proportion of results chosen for publication are unrepresentative of scientists’ repeated samplings of the real world. The self-correcting mechanism in science is retarded by the extreme imbalance between the abundance of supply (the output of basic science laboratories and clinical investigations) and the increasingly limited venues for publication (journals with sufficiently high impact). This system would be expected intrinsically to lead to the misallocation of resources. The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality. Science is subject to great uncertainty: we cannot be confident now which efforts will ultimately yield worthwhile achievements. However, the current system abdicates to a small number of intermediates an authoritative prescience to anticipate a highly unpredictable future. In considering society’s expectations and our own goals as scientists, we believe that there is a moral imperative to reconsider how scientific data are judged and disseminated.
Of course, the authors are part of the same system they criticize, so if this gets cited a lot, prepare for it to be debunked…
(I’m also a little leery of the original claim, which doesn’t really seem to address what you need to know — fer example, how wrong are the famous wrong papers, since plenty of revisions don’t affect your central thesis, how does this wrongness compare to the wrongness of nonfamous papers, and how much of the difference, if any, is the result of many more people bothering to examine the famous papers, while the nonfamous ones don’t even get examined for accuracy.)
The line about selectivity is also a bit misguided — it’s not about the cost of print, it’s about the cost of attention. Selectivity is there because people can’t keep up with even a couple journals in their own field, much less everything that would like to be published. (or maybe those are my editorial biases talking)