A group of scientists thought they’d take 100 experiments published in the top journals in 2008 and re-perform them. They wanted to see whether the results would reappear.
What they found seems to have sent them into a collective depression.
The study, published Friday in Science magazine as “Estimating the reproducibility of psychological science,” concluded that psychological science is confirmed relatively rarely.
Indeed, only 36 percent of the experiments produced the same results. Well, when I say “the same,” what I mean is results in the same direction, but on average half the effect that was shown in the original study.
The 270 scientists involved in this deeply realistic project discovered that 75 percent of the studies in social psychology weren’t replicated. They also re-examined those in cognitive psychology and found that around half of those produced very different results.
Here’s how somber this new study is: “A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes.”
The researchers were led by Professor Brian Nosek of the University of Virginia. He told the Guardian he was disappointed “in the sense that we could do better.” There is another sense?
He added, specifically for you, dear reader: “The key caution that an average reader should take away is any one study is not going to be the last word. Science is a process of uncertainty reduction, and no one study is almost ever a definitive result on its own.”
This is manifestly true. However, the process of modern science allows for some troubling fears. As the research report itself says: “Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication.”
Buried within these words is, perhaps, the idea that to rise in their profession, scientists need to publish — a lot. If they publish something newsworthy — in whatever way you wish to define “news” — they are again more likely to garner attention.
So they move from one “interesting” experiment to the next. In moving so quickly, each experiment is small, a quick hit rather than a larger-scale study with the potential for a more universal conclusion.
The research report looks scientific journals’ role in this process squarely in the soul too: “Innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that ‘we already know this’ belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both.”
We rarely know as much as we think we do. Talk to many scientists over a hamburger and a margarita and, at some point, you might hear the apparently knowledge-laden phrase “studies show.”
What and how much do they really show?
Science is ultimately very, very hard. Making it concrete clearly involves replication, especially in a field like psychology which deals with something as elusive as the human mind.
Of course, these results don’t prove that the experiments they tried to replicate were fake — or that they were poorly conducted. They could be a false positive, or a minute change in research conditions might have affected the results. Might have.
I want to end with a little hope. Let’s talk about one study that was replicated. This showed that people have enormous talent at recognizing expressions of pride in different cultures.
It’s a tough one, pride.
It keeps human beings alive when all things seems to be going against them. It also comes before a fall. Well, some of the time. What percentage of the time, I wonder? I think we need an experiment to find out.