The Replication Crisis (Still)

Despite Positive Changes the Past Two Decades Crucial Structural Issues Remain

The great thing about science was always alleged to be its impartiality. It cared about facts, and wasn't above making a mistake, which is to say, hypotheses are built to be checked, not necessarily confirmed. But academia is full of human beings, which are known to lie and exaggerate, particularly when something valuable like prestige, a job, grants, and possibly a book deal could be on the line.

Obviously this is a structural incentive, and the phrase “publish or perish” was coined by a sociologist in 1942, so there's nothing new in that. However it could be that the sheer quantity of college throughput is so much greater than it was then, that perhaps the environmental pressures are greater.

In 1940 there were approximately 150,000 people employed in higher ed working at 1708 institutions serving about 1.5 millions students. By 2010 that was 4 million employees serving 21 million students at over 7000 degree-granting institutions. That's a more than 25-fold employment increase in seven decades. That implies a lot more middling talent trying to distinguish themselves to survive.

These kind of pressures are fingered as the primary culprit in a two-decade long odyssey known was the Replication Crisis that's cut a wide wake across the science, since Stanford School of Medicine professor John Ionnidas published the 2005 essay, “Why Most Published Research Findings Are False.” Ionnidas argued that a large number, if not the majority, of published medical research could not be replicated.

This kicked off a burst of replication studies. One of the reasons people could get away with publishing research of such questionable quality is because replication studies were rarely printed in journals creating little incentive to make the effort. One study found that 1% of the articles published in major publications were replications. The lack of oversight is a huge moral hazard.

That burst of study found that Ionnidas was right. While he was focused on medical research, initially softer sciences like psychology and sociology were hit the hardest, perhaps due to less experience in rigorously structured research studies. An influential effort called the Open Science Collaboration found that of 100 independent replications only 39% were successful and on average the effects were half the size of the original paper.

The White Lie of Academia?

Ionnidas' argument in part centered on what would later be called p-value hacking or simply p-hacking, the practice of manipulating the data in a variety of manners until you create a statistical significant result, which is when P<0.05, which is often understood to mean that there's a 5% chance of a false positive. However there are a great many ways that you can “game” the results or the way you collect them. One paper noted a dozen different strategies people use – you'd think they were trying to beat Vegas at blackjack. Then again, that might not be so off.

Dr. Roy Baumeister, a social psychology professor at Florida State University coined the theory of ego depletion in 1998. It suggested willpower was a muscle that could be depleted by work. While middle managers around the globe may have rejoiced, their formidable new musculature was never replicated, despite a fury of attempts.

A whole field of study erupted around Baumeister & Co.'s findings, and he wrote a New York Times bestseller with NYT writer John Tierney in 2011, Willpower: Rediscovering the Greatest Human Strength. Did the questions about his theory stop him? No, Tierney & Baumeister published another book, The Power of Bad* in 2019. (The Guardian panned it, describing them as "pro pollyannas" boosting a "complacent, reactionary book." Well, where does Tierney work?)

This after a 2016 attempt in two dozen labs, involving 2141 participants that failed to find any evidence of the effect.

A Dutch social-psychologist Diederik Staple was found to fabricate or manipulate data in 55 studies. The famous marshmallow test where a 4-5 year-old child's ability to resist eating a marshmallow when left alone was alleged to strongly predict adolescent achievement didn't replicate. Neither did listening to Mozart make you momentarily smarter, nor any of the research on priming with words or “power” poses.

But the problem is so much deeper than the black box of our minds.

Bayer Healthcare attempted to replicate 67 influential drug studies succeeding with 14, or 21%. A 2012 study by Amgen found only 11% of landmark cancer findings could be replicated. That should not surprise anyone closely following the number of foods that have been said to cause cancer.

Perhaps that explains why everything causes cancer.

The pressure is up and down the line. In 2024 the prestigious Dana Farber Cancer Institute, an affiliate of Harvard Medical School, retracted 6 studies and offered corrections on 31 more as part of a review of four researchers who falsified data. Another researcher's fabrications prompted Northwestern University to pay back $2.3 million in grant monies.

It's proven pervasive across every field and status level. A 2015 study the Federal reserve of 59 publications in 13 influential economics journals could only replicate half, even with the assistance of the original authors.

Even Worse Than Just Lying

The journals are complicit. Publications want papers, and are in competition to publish noteworthy research. They and the researcher's educational institutions aren't interested in looking that closely. Nor are they particularly interested in reviewing their own mistakes.

It took The Lancet twelve years to withdraw Andrew Wakefield's 1998 fraudulentt paper linking the MMR vaccine to autism. This despite a 2004 article revealed conflicts of interest – that Wakefield had been hired by a law firm aiming to sue vaccine manufacturers two years before he published the study. Or the fact that Wakefield had filed a patent on a competing vaccine.

Lancet retracted the article in 2010. In January 2011 a journalist who spent years looking at the data published an article in the British Medical Journal charging Wakefield with altering/misrepresenting the data. Wakefield lost his case and his license to practice medicine in England. It only took 12 years, six since his deep, and hidden conflict of interest was revealed.

The editor, Dr. Richard Horton, who refused to retract the paper, offered a classic, "I'm just sorry people feel bad" (eliding over his role in classic 21st century elite manner), saying "I regret hugely the adverse impact this paper has had.... [but] Professionally, I don't regret it. The Lancet must raise new ideas.” (This seem to be the New York Times approach to covering our president. It's "like news" in that there's information in sentence form, but it's not accurate in amy real sense of word.)

Sixteen years later, all of America is still paying for Horton's dubious ethics. In 2023 he was named Officer of the Most Excellent Order of the British Empire, the fourth highest honor one can receive from the figurehead monarchy. Consequences, amirite?

Another big case involves faked Alzheimer's research by Eliezer Masliahm for ten years the head of the Division of Neuroscience in the National Institute on Aging. Another Alzheimer's researcher at the University of Minnesota was implicated in more than 20 papers, including a 2006 one identifying a particular plaque-forming amyloid beta protein, leading researchers down a blind dead-end for almost twenty years

Once published, bad research is treated identical to the good stuff. One study found unreliable research tends to be cited as if the results were true long after the publication failed to replicate. Further, two different studies in 2020 and 2021 found papers with reproducible results tended to be cited as often or less than those with findings that leading publications could not reproduce. They suggested the publications face a trade-off between maintaining standards and how exciting the results are.

Often, It's No Surprise

One begins to feel the force of these badly aligned incentives when you look at the quality of the papers. While some of the time it's difficult to detect fraud in the data, much more often it's about poorly designed studies, with small sample sizes, a surprisingly large effect that might look weird in the cross-tabs (see again, sample size) and with a possibility of the effect showing up by chance, right at the p-hacking border. Any undergrad could suss this out.

Indeed, the things Ionnidas cites as warning signs of a weak, or potentially unreproduceable study are pretty intuitive if you've had instruction in science and methods. One study of social science papers found that laypersons could tell nearly 60% of the time, “ on the basis of nothing more than simple verbal study descriptions.”

Academics are even more savvy to what a good study looks like. A 2018 Nature study utilizing a betting market found scientists were very accurate at guessing which psychology studies would replicate.

“We also know that experts can predict well which papers will be replicated,” write the authors Marta Serra-Garcia, assistant professor of economics and strategy at the Rady School and Uri Gneezy, professor of behavioral economics also at the Rady School. “Given this prediction, we ask ‘why are non-replicable papers accepted for publication in the first place?’”

The issue is largely that the incentive structure is still mostly unchanged. UK plant biologist Ottoline Leyser has argued, “The reproducibility crisis is actually a publication bias crisis which is driven by the reward structures in the research system.”

Those reward structures aren't changing, leaving most of the work on the replication crisis to happen around the edges.

Low Hanging Fruit

One of the good things to come out of the replication crisis is the crisis itself. Suddenly doing a study that fails to replicate a famous study is much more sexy. “A scientist can make a big splash by discovering that a famous result cannot be replicated,” says statistician Larry Hedges, a fellow at Institute for Policy Research.

One advance is something known as open or pre-registration, where researchers share their hypotheses and methodologies as a safeguard against later changes, often made to fix the data by changing the parameters. This is voluntary and probably doesn't do a lot to discourage bad behavior by the incorrigible, but probably stops some late-night-crisis-of-the-soul sell-outs.

“As somebody who's been preregistering for years now, I appreciate how much extra work it is,” psychologist Dr. Jennifer Tackett said. “But I also appreciate how much it really does keep you from fishing around or fooling yourself into thinking you did things a certain way, after the fact, you know, when you have to really put things down in advance.”

Another new approach is the multisite collaborative research, which involves doing the same study at two different institutions. While the only evidence we saw was anecdotal, this approach has apparently swelled as researchers look for safety in numbers - larger sample sizes, more independently involved labs.

Hedges has gone on to suggest that the problem is less one of replicability and reproducibility. “Most scientists now distinguish reproducibility (Can you get the same answers that I did when you analyze my data?) from replicability (Can you get the same answers that I did when you do my experiment and collect your own data?),” Hedges said.

However, that may just be the kind of coddling that got us here.

A six-year collaboration between labs at UC Santa Barbara, UC Berkeley, Stanford, and the University of Virginia that used preregistration, strong methods and large sample sizes was able to replicate 16 novel findings at 86%, and with 97% of the effect. As noted earlier, many replication studies fail to achieve even half the original studies effect.

The 2023 study suggests that with rigorousness adherence to established standards replication is quite possible. What we're witnessing is precisely what people think we're witnessing, and the impulse to naysay and soft-soap the truth is part of how we got here. [This study was subsequently retracted over lack of suggested preregistration and questions about fitting the data to the analysis.]

So while are suggestions in some fields that they've made standards more rigorous, and that p-hacking is declining as journals get more serious about fringe-y studies, any suggestions of improvement must be accompanied with a grain of salt. The people who didn't care about replicability are the same ones who will hand wave the issues away because the most troubling parts are cooked in.

People Do People Things, and Too Often That's Bad

Duke postdoc Paul Bogdan looked at the p-values that measure for significance in a 2025 paper published in Advances in Methods and Practices in Psychological Science. As mentioned, studies general need a p-value of 0.01 to 0.05 to be considered significant. (Significance is measured as a 1%-5% chance the findings occurred at random.) He looked at papers from 2004 to 2024 and found the share of "fragile" significant results dropped from 32% at the start of the crisis to 26% recently, and found the shift among every discipline and sub-discipline.

It was driven largely by increased sample sizes, which also seemed to push the effect down slightly. Small studies inflate the effects of their findings whereas bigger ones give truer, smaller estimates. Interestingly, Bogdan found scientists at top-ranked universities published slightly shakier numbers. One wonders if that might be because the pressure to publish in better journals is at its most fierce at those institutions.

On the other hand a 20% drop in fragile submissions is not the response to a "crisis," it's more like CYA (e.g. Cover Your Ass). We're still talking about a quarter of the papers have sketchy significance. That's a new seating arrangement for a sinking ship.

A 2025 paper for the Journal of Academic Ethics by Yasemin J. Erden at University of Twente, in Enschede, Netherlands echoes our initial, "intrinsic problem" concerns. Erden points to a more quotidian explanation than "poor methods/statistics background," Hyper-ambition and the Replication Crisis.

"There are already many proposals to address questionable research practices, some of which focus on the values, norms, and motivations of researchers and institutes, and suggest measures to promote research integrity," Erden writes.

"Yet it is not easy to promote integrity in hyper-competitive academic environments that value high levels of ambition," he continues. "It is as likely that a kind of hyper-ambition is fostered that (inadvertently or otherwise) prioritizes individual success above all, including to the detriment of scientific quality.

"In addition, efforts to promote values like integrity falter because they rely on sufficient uniformity in motivations or tendencies," writes Erden, inserting the knife. "Codes and guidance promoting integrity are, however, likely to influence those for whom such values are not optional, while others simply find ways around them."

In other words, cheaters are gonna cheat, and giving them classes about being good people and behaving properly isn't going to help those who value their own welfare above all else.

The real "black box" human behavior question we won't get an accurate answer to is how many people in our system believe it's worth the risk, because the chances of exposure are vanishingly small, consequences all but non-existent and the reward.... well it's reward, and if you're the type that does this, the size of the reward is whatever your mind needs it to be to rationalize your behavior.

The truth, they say, will set you free, but I suspect it's more likely to sap your faith. Living is learning.