On Peer Review
Savage Minds reminds me of Ed Carr’s commentary on peer review from late December. (Carr is a geographer who I interviewed recently for Yale Environment 360.) Here is a provocative excerpt from his post:
I have found peer review to often function as a means of policing new ideas, slowing the flow of innovative ideas into academia not because the ideas are unsupported, but because these ideas and findings run contrary to previously-accepted ideas upon which many reviewers might have done their work. This byzantine politics of peer review is not well-understood by those outside the academic tent, and does little to improve our public image.
I can’t speak to this, since I don’t publish peer reviewed articles. Anecdotally, I have heard complaints of groupthink from some anthropologists and archaeologists. Readers of this blog who are keyed into the climate wars will naturally project their own biases onto Carr’s experience. But I seriously doubt that he had climate science in mind when he was writing his post. Rather, his commentary is aimed at breaking down the “walls of academia,” in general.
“Anecdotally, I have heard complaints of groupthink from some anthropologists and archaeologists.”
Funny. To put it extremely mildly.
Two of the most classic examples of ‘Consensus’ groupthink are from this corner of science.
First, who can forget Piltdown Man? Classic example of a false conclusion supported by a political agenda.
Second, the ‘Clovis First’ debacle. When the group all agreed that the whole colonization of North America was simply explained, and then stuck with it despite mounting contrary evidence… and going as far as to deliberately ignore/deny that evidence and smear those who supported it. Was almost a test run for the AGW project.
The quality and value of peer review depends entirely on the knowledge and OBJECTIVITY of the reviewers, and of course, those who control the journals.
The main problem with peer review is that the people doing it are all just humans, and all humans are political and economic animals prone to groupthink.
Carr: Anything that survives peer review is by and large more reliable than an unvetted website (like this one, for example).
Carr appears to have social sciences in mind specifically:
I will note, though, that I say this from the perspective of a qualitative social scientist working mostly in the area of social theory. In this area, we have a lot of theoretical debate that is not easily resolved via reference to empirical evidence
I’m not making any conclusions about Carr, but those who submit junk coincidentally make a similar charge, although it’s more of a way to cover their behinds and present the illusion that they are just being persecuted by the close-minded establishment for presenting the “real” truth. Such spin works in climate science since there’s an audience for it. It’s Roy Spencer’s perpetual whine, for example. In other cases, peer review often isn’t sufficient at preventing junk papers (that also happen to go against established views) from being published. McLean’s piece on claims of ENSO and warming come to mind, as does the Soon/Baliunus debacle where submitters targeted a journal with an editor know to be sympathetic to their political cause. These papers aren’t difficult for an astute observer to note the fatal flaws in. Like Ed says, the quality and objectivity of reviewers is important. Unlike social science, bricking basic statistics and misusing data isn’t a matter of subjective debate.
But Carr’s overall view of peer review being by in large more reliable than the altnernative is correct.
Carr’s comments are in a similar vein as Keith Seitter’s column highlighted by Judith Curry
http://judithcurry.com/2012/01/28/keith-seitter-on-the-uncertainty-monster/
As he noted, there are scientists who are convinced that humans are affecting climate in significant ways but who feel that anthropogenic influences other than the increase in greenhouse gases, such as aerosols, land use changes, etc., can play a larger role than typically acknowledged. The scientists studying these other human influences “” despite being among those I would refer to as among the “convinced” “” sometimes find their work discounted, or even marginalized, since their results complicate the simpler picture of increasing greenhouse gases representing the only major anthropogenic forcing term for a changing climate.
The sentence immediately preceding Jeff’s highlight:
Shortly after that column appeared, I received a note from a long-time AMS member who rightly suggested that I had overly simplified the situation.
Speculating of course, methinks that note is just Pielke Sr., long-time AMS member, who typically presents this argument as a way to mimimize GHG, rhetorically at least. There was a long back-and-forth at SkS on the topic, that Pielke engaged in (dodginess and all) But regardless, whether he wrote the note or not isn’t important. It also doesn’t seem relevant to peer review. Pielke, for example, has published extensively, and on the topic of UHI and LULC changes, even erroneous or subsequently corrected work.
Hansen, Ramanathan, and others have published extensively on the significant effects of aerosols, both sulfates (negative forcing) and black carbon (positive forcing) on global climate. See for example
Novakov et al, 2003, “Large historical changes of fossil-fuel black carbon aerosols”
I doubt they consider themselves marginalized, and I suspect they welcome critical review.
Good scientists and skeptics welcome scrutiny from their peers, seeing it as an opportunity to improve on their work, or discover something they hadn’t previously considered. Climate “skeptics” unfortunately don’t always engage in good faith efforts to advance science.
I think Seitter should have looked a little more closely at the note he received, rather than being too diplomatic in stating it without critical review. Curry and Webster’s work (also published) also deserves a bit of scrutiny, and a comment has been published on their article (Hegerl et al.). Their claims have some basic logical disconnects, some of which are not difficult for a layperson to spot.
http://www.realclimate.org/index.php/archives/2012/01/the-ar4-attribution-statement/
That is why they have conferences. Believe Eli you can put any dumb idea you want onto a poster, or even an invited talk and some have (see AGU: Spencer, Roy.
A bit OT, but anyone interested in following up NJY’s remark about the Pielke – SkS exchange may be interested in the SkS summary of RPSnr’s under-representation of the role of CO2 in recent warming.
Not exactly peer review, but valuable and informative all the same. I say this as someone formerly persuaded by RPSnr’s arguments.
If peer review shuts the door on unconventional ideas, please explain this.
The Carl Sagan effect may be at work in many complaints about closed-minded peer reviewers.
Jonathan,
There will always be exceptions, or such examples, no? Does that negate the fact that some amount of policing exists? I don’t think so.
I have found peer review to often function as a means of policing new ideas
Pretty much any system that seeks to enforce ‘truth’ will occasionally error and end up filtering out truth even if it doesn’t suffer from group think or corruption.
Even fair minded editors of non-scientific publications struggle with how much space should be dedicated to the rantings of the ‘village crank’.
On the contrary, many journals face pressure to publish new and interesting things. While I don’t think most cross the line into the paranormal, there’s a strong desire particularly in climate science to overturn well-established views, and occasionally crank ideas are given a little too much leeway. I don’t think this is unique to climate science (as JG’s first link in #7 indicates), but amplified by the demand from certain elements of society for it. But on the whole, peer review works quite well – far better than the shoddy stuff that is plastered around the blogosphere, at Monckton rallies, and among anti-science “think tanks”.
The idea that peer-review functions to enforce ‘truth’ is a fundamental misunderstanding of its purpose. The purpose of peer-review is to decide whether a paper is worth your time to look at.
Journals are businesses out to make money by providing a service, and that service is to collate developments of interest to their target audience, so that busy scientists can keep up with the state of the art in their field with a minimum of effort. They review submissions to see if they are of interest, peer-review being used if the in-house review is insufficient, and it is usually the case that readers won’t want to read papers that are obviously wrong. But it’s rare for any real testing of the claims to be done.
However, testing correctness is not the primary purpose of peer-review, and journals have sometimes published papers in the belief that they are wrong, making the argument that despite this they are of sufficient interest to merit people’s attention for other reasons. The FTL neutrino experiment is a good example of that.
The test of correctness comes when the readers of the paper try to replicate the result – checking the calculations, re-doing the experiments, testing, extending, falsifying. Until this has been done, papers have to be considered unverified hypotheses. The journals consist of work in progress, unless you happen to know which of them have been subsequently confirmed.
That’s the theory, anyway. In practice, two major factors distort this nice picture. The first is the “publish-or-perish” use of publication indexes (papers, citations, impact, etc.) as a measure of performance for pay, funding, and career. This leads to pressure to get published at any cost, and journals to cater for that need. The second is that people with opinions and a strong stomach for bureaucracy can take control of funding committees and editorial boards to influence the field to their advantage. People play politics. In theory, the government is about running the state well for the benefit of the people, but in practice it is about lobbying and earmarks and bribes and ‘influence’. Accademics are human, too.
Some journals do and some don’t. We can’t generalise from the behaviour of journals in one field to every other field.
Just because it’s evidently expected, I’ll mention the climate debate’s bad examples: “If published as is, this paper could really do some damage. It is also an ugly paper to review because it is rather mathematical, with a lot of Box-Jenkins stuff in it. It won’t be easy to dismiss out of hand as the math appears to be correct theoretically”. But we all know the script by now. Ppeople are outraged that papers with horrific errors have got past review when they’re supportive of the consensus, and have faced a far harder time when they’re not. And conversely, many are horrified that, despite all their efforts, some papers critical of the consensus have got past their guard.
The question of which set of papers is correct (everyone believes their own side to be correct, of course) is a completely different issue to the question of whether people who are not correct should nevertheless be allowed to get published.
Given the real function of peer-reviewed journals, the answer is obviously: yes, if there’s a market for reading them. But for those who see themselves as gatekeepers, it’s a difficult balance between risking the purity of the field and giving the appearance of censorship.
Personally, I’m all in favour of an occasional paper on ESP being published. If you can’t knock it down, pronto, there’s something up with your testing methods, and that’s worth knowing. ESP is rejected because it’s been proven to be wrong, not because nobody dares to publish research on it. It’s only the fact that the door is unlocked that makes their rejection credible.
That it upsets the gatekeepers is a bonus.
@Keith. Of course. My point is to ask which is the rarer example, the wacky thing that gets in print, or the good work that’s suppressed (e.g., Boris Belousov). The blog at your link gives a subjective impression (the author feels there’s a lot of policing) without giving evidence that this is a big problem:
“The proliferation of published work that has emerged from these two trends has not really improved the quality of information or the pace of advances ““ there is still a lot of good work out there, but it is harder and harder to find in an ever-growing pile of average and even not-so-good work. And I have found peer review to often function as a means of policing new ideas, slowing the flow of innovative ideas into academia”
Where is his evidence that opening the literature to more “innovative” work would result in more substance, as opposed to more dreck? Hence my reference to Sagan’s “Bozo the clown” line.
Andrew Gelman has some interesting commentary on his perception that peer review does too little policing, not too much.
Jonathan, on a related note, see this in Nature.
I’ve linked to the following study so many times it may as well be about my theory of the Iron Sun:
Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System
http://people.stern.nyu.edu/wstarbuc/Writing/Prejud.htm