The Journal Nature Requires Reproducibility In Submitted Papers
“Replication is something scientists should be thinking about before they write the paper,” says Ritu Dhand, the editorial director at Nature.
Now this will stop a lot of AGW crap from being spewed... Requiring all data and methods be disclosed is going to stop a lot of bullshit from being published. No more hiding the data or refusing to disclose your methods.. Finally!!! Real science might get accomplished and some integrity returned to science..
I have to say it's not altogether clear to me why this topic is in the Environment forum rather than, say, Science/Technology, Education, or perhaps Structured Debate. The thread's presence here in such a narrowly focused forum, given that
Nature publishes papers covering the entirety of science disciplines, gives me pause insofar as I cannot help but sense the purpose of the thread is not to address inadequacies in the nature of
Nature's published articles, but rather it exists to push an agenda pertaining to the climate change debate.
If the thread and OP-er's
raison d'etre be that of advancing/advocating what appears (to me) to be a calumny theory premised upon notions of collusion of some set of actors in the debate over the politics of climate change, I don't really have something I care to share. I have nothing substantive to contribute in that regard not because I don't understand the matter, and not because I don't concur about the importance of reproducibility, for I absolutely do so concur. I don't because I know the futility of bothering here to engage in a climate change discussion. In a less petty environment, I might actively share my thoughts, or at least pose some thoughtful questions that may help advance the discussion.
If, on the other hand, the discussion theme is intended to be one of the technical merit of papers
Nature (and by extension, competing science journals) publishes, I have the thoughts below to share as goes the peer review process.
integrity returned to science..
Before getting started, I have to say I think science and scientists have a good deal of integrity, but I'm not naive about the fact that there are lapses, material ones even at times. But that, I think, is a related but different discussion topic, so I will move on.
The current system of peer review is not perfect, and, while most scientists believe it is necessary, indeed, desirable, the core assumptions inherent in the process must be evaluated and adapted to the changing environment. While research has examined modifications to the process, biases, and other flaws, perhaps the purpose of peer review and some key assumptions should be examined. For example:
- Should manuscript peer review emphasize the validity of the science or its merit? While peer review is often viewed as the gatekeeper to the realm of truth, a staff of editors and reviewers cannot make that distinction in a few months.
- In terms of interesting assumptions overlooked, does the process of peer review require a true peer or expert in the field?
- Would unfettered publishing of findings lead to more efficient or faster progress in science?
- What does the stamp of peer review mean to the readers of the literature?
I think
Nature's editorial board have content and intellectual integrity constantly in mind. In the mid-aughts and in response to suggestions from readers and contributors,
Nature explored a new approach to its peer review process, the so-called open peer review approach. (Click the link to learn what the tested process entailed.) It found that while "open" reviewers contributed some editorially relevant feedback, very little technical input and critique resulted from the open review approach. The journal thus opted not to implement the open review process.
In light of
Nature's sincere efforts to boost the qualitative value of the peer review process, I find that the publication's editors understand the importance of integrity. That said, they are science editors, not experts on every science topic about which they may receive submissions. As such they depend on the input of subject matter experts to participate in the review process. When that participation is lacking, as it was in the noted open review trial, what are the editors to do?
Now the editors are trying to force one (several) peer reviewers to perform replication exercises. Okay. It's a different approach and a new standard that manuscript authors must meet. Like the "standard" peer review processes, it too has pros and cons. Pragmatically speaking, the replication obligation is for some manuscripts fitting and for others, well, not so much.
- Should a researcher who's made a preliminary discovery need to meet the requirement? Often scientists come by ideas they explore incompletely, but they publish papers about the idea for the sake of engendering further research into the matter. Essentially they do so to say, "Hey. I just noticed 'this' and looked into it to the following extent...What I found suggests XYZ and I think it'd be worth looking into further." What such researchers observed/examined could well be an anomaly, or it could be that it's not an anomaly but not a frequent or reproducible-at-will thing. The natural world operates under a set of rules, but it's not software either.
- Should prohibitively expensive experiments also be reproducible? Ditto re: data collection? Sometimes what a scientist publishes is the story of an experiment s/he performed for the purpose of collecting data. For instance, a physicist researching gamma ray bursts is constrained by the mere occurrence of the event. Should her paper not be published because nobody else has seen another burst, or because no burst has yet occurred?
What I'm getting at is that scholarly publication is scholars' way of debating. One researchers says, "This is what I've seen and what I conclude from it." Another takes the idea and explores it from a different angle and publishes to say, "I accept part of what you say, but I looked into it 'this' way and found ABC, which it think attenuates or invalidates your conclusion." The next guy, having read both papers performs his own research inspired by the "debate" and shows how one of them is "right" given X-weighting of the factors in play, while the other is "right" given Y-weighting of them. Yet a fourth examiner looks into it and finds that there is a new factors that none of the earlier researchers noticed or considered and show that element to be the thing that defines the cause of the varying results earlier researchers found.
In short, scholarly publication is as much about the specific technical findings as it about the process of expanding a discipline's body of knowledge. The process may not go as quickly as some might like. It may be imperfect, and
Nature seems to be attempting to mitigate an aspect of the imperfection. I just hope they don't go too far with the replication "thing" that by enforcing unreasonable demands that squelch idea expressions they hinder scientists' ability to have, as they do now via the paper publication process, widely accessible discussions that inspire among others innovative thought .
