What is Peer Review And How It Can Go Wrong?

To paraphrase Winston Churchill, scientific peer review is the worst system in the world … except for all the rest. (Scroll to the bottom for his exact quote which applied to democracy.)

In short, peer review, when conducted honestly, requires a scientific paper being considered for publication to be submitted to: a panel of experienced, unbiased scientists who are knowledgeable about the topics in the paper.

That panel makes comments about the way the experiment was designed, the protocols involved, the quality of the data, whether the data supports the papers conclusions and other factors that determine whether the paper is suitable for publication.

In addition, properly selected peer reviewers often know of undisclosed biases and conflicts of interest among paper authors that need further disclosure.

Often a paper is edited or re-written to address comments before publication. Sometimes the flaws are too severe and the paper is rejected. Sadly, the rejected paper will eventually be published by a journal with sketchy ethics.

But, when it works correctly, peer review helps assure the overall quality of the article. Today, that has been combined with with more recent requirements by top scientific journals that papers include complete information sufficient for others to reproduce the experiment and results.

It addition to better science, this process helps protect a scientific journal’s reputation and credibility.

Systemic Flaws in Peer Review

As Churchill hinted, democracy is a messy system with systemic flaws. Peer review is no different.

Peer review has flaws, but it remains the best system to control the quality of science.

Sadly, corporate, private and regulatory science lacks any quality control system at all. See “Why You Can’t Trust Corporate & Federal Regulatory Science” for details.

Also: Regulatory agencies must demand conflict-of-interest statements for the research they use.

Unpaid Reviewers

The most serious systemic problem facing the peer review system is that scientific journals who charge hundreds and even thousands of dollars for an annual subscription pay their peer reviewers nothing.

Some reviewers work for free so they can get into the good graces of editors in hopes of getting their own papers reviewed.

Often, the most qualified reviewers are simply too busy with their work and cannot afford the time or the financial sacrifice to review another paper.

See:

Reviewer Selection

Perhaps an even worst systemic flaw is the practice of journal editors asking a prospective author for a list of potential reviewers. Few scientists would stack that list with their enemies.

That can produce the same results as a crooked science fair

Conflict Of Interest Disclosure

While respectable journals require conflict of interest disclosures of authors, they do not require the same of reviewers.

In addition, journals use a very narrow definition of “conflict of interest” to mean a direct financial interest.

Neither reviewers nor journal authors are required to disclose previous associations such as employment or consulting positions with chemical corporations, tobacco companies, pharmaceutical and medical manufacturers, private laboratories, advocacy and leadership positions with industry groups and other parties which have direct financial interests in the outcome of studies and their conclusions.

A Major Fraudulent Flaw

While the narrow definition of “conflict of interest” is an inherent problem that journals are trying to address, outright deception and fraud are possible.  Examples of this can be found at:

In addition, another example of fraud that is being exposed lies in networks of fake reviewers — people who simply do not exist. A number of those have recently resulted in the retraction of scores of scientific papers. (See:  Scholarly journal retracts 60 articles, smashes “peer review ring”)

According to the Washington Post: “a major publisher of scholarly medical and science articles has retracted 43 papers because of “fabricated” peer reviews amid signs of a broader fake peer review racket affecting many more publications.

[T]he Committee on Publication Ethics, a multidisciplinary group that includes more than 9,000 journal editors, issued a statement suggesting a much broader potential problem. The committee, it said, “has become aware of systematic, inappropriate attempts to manipulate the peer review processes of several journals across different publishers.” Those journals are now reviewing manuscripts to determine how many may need to be retracted, it said.

 Keeping An Eye On Sketchy Science

Retraction Watch is a magnificient source of information on sketchy science.

Relevant Links

  • The Problem of Biased Data and Potential Solutions for Health and Environmental Assessments

 


From: Respectfully Quoted: A Dictionary of Quotations.

Sir Winston Leonard Spencer Churchill (1874–1965)
QUOTATION: Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.
ATTRIBUTION: WINSTON CHURCHILL, speech, House of Commons, November 11, 1947.—Winston S. Churchill: His Complete Speeches, 1897–1963, ed. Robert Rhodes James, vol. 7, p. 7566 (1974).
Comments are closed.