The gist for this post came from:
“Research, Report, Repeat“– an article from Discover Magazine (Yong 1/14).
The story interviews psychologist Brian Nosek, at the University of Virginia, who is at the forefront of the fight to make psychology more transparent. While you’re reading this, delete in your mind the word, “fraudulent” and put in its place “incomplete.” In place of “charlatan” put “traditionalist” in its place.
From the article:
“In 2011, Dutch social psychologist Diederik Stapel was revealed as a charlatan who had published dozens of fraudulent scientific papers. The shocking thing was that no one in his field had noticed until courageous students in his lab reported their suspicions. This spate of misconduct happened in tandem with many failed attempts to replicate some of the field’s classic results, prompting acolytes to question whether psychology was being polluted by quirky and attention-grabbing findings that might not actually be true. This issue applies to every field of science, from physics to medicine.“
The key point to the article is reproducibility.
Hypotheses, phylogenetic trees, reconstructions that fit footprints—all of these things in paleontology must be reproducible and tenable. Unfortunately, as I’ve reported here, replication often fails, typically due to an incomplete inclusion set. Sometimes, as in deep chord wing membrane papers, to wishful thinking.
We’ve seen this in phylogeny following attempts at nesting pterosaurs with archosaurs, at nesting mesosaurs with pareiasaurs, at nesting caseasaurs with pelycosaurs. All that was fine, but now that the repair is available (see www.reptileevolution.com) there is less excuse for continuing phylogenetic errors based on incomplete inclusion sets.
The pterosaur heresies blog is here to reproduce and verify or falsify the work of other paleontologists. Just as other paleontologists are doing all the time.
Testing is key.
Did they include the proper inclusion set? Too often they did not. Did they find sisters that provide a tenable evolutionary path that provides a gradual accumulation of traits? Too often they did not. Did they invent excuses for soft tissue not behaving in situ according to their traditional hypotheses? Too often this is so. Did they really try to pull details out of the specimen or did they draw a crude circle around the crushed skull and call it a day?
all these were done with the best intentions following established procedure with no intention to defraud. Rarely, as in Bennett’s 2008 imaginary protopterosaur, hypotheses can appear that are completely imaginary with no basis in reality, no thought to untenable evolutionary scenarios, and no testing or review of prior published studies and specimens.
Back to the Discover article:
In 2011 Brian Nosek and colleagues launched the Reproducibility Project, in which a team of about 200 scientists are carrying out experiments to replicate findings published in psychology journals. They want to examine the problem of false-positive results.
Nosek reports, “There is an ethic that science is self-correcting, but replication is often a pain the in butt, and since scietists’ career success doesn’t depend on exactly replicating a study that’s already been published, they usually don’t do it. As a result, mistakes end up hanging around longer than they need to.”
The same can be said of paleontology.
Most of paleontology is on an even track. But there are still problems out there, like “modular evolution” and the exclusion of tiny (sparrow-sized) pterosaurs and fenestrasaurs from analyses.
Nosek built the “Open Science Framework” a web application where collaborating researchers can put all their data and research materials so anyone can easily see them. The same thing is happening for paleontology at ResearchGate.org and Morphobank.org.
Nosek concludes, “Replications are so rare that people perceive them to indicate a lack of trust. I want it to be very ordinary for people to say, “I’m not sure about this, so I’ll replicate it,” and for that to be a compliment, not a threat.”
I heartily agree.