WHEN Stuart Ritchie was a graduate student in Edinburgh, UK, in 2011, he was involved in an incident that shook his faith in science. With two colleagues, he tried and failed to replicate a famous experiment on precognition, the ability to see the future. They sent their results to the journal that published the original research and received an immediate rejection on the grounds that the journal didn’t accept studies that repeated previous experiments.
Ritchie remained a scientist – he is a psychologist at King’s College London with a focus on studying human intelligence – but ever since that rejection, he has been on a crusade to air science’s dirty laundry. His latest book is , in which he shows how, all too often, we can’t rely on the facts that science provides.
Advertisement
Graham Lawton: The grand and scary claim of your book is that something is rotten in the kingdom of science.
Stuart Ritchie: Absolutely. We think of science as being this objective thing that tells us facts about the world and produces all these scientific papers, which are almost sacred things. But a lot of people don’t see how the sausage is made. I think if they had more of an idea of how the process happens, they would question the truth status of those papers much more. In a lot of cases, the science is useless, not worth the paper it is written on.
You identify four main causes of rot.
First there’s fraud, when people deliberately alter or make up results to try to get a paper published. That’s rare, but not as rare as we would like to think. If you ask scientists whether they’ve committed fraud, only a very small number say yes. But something like 14 per cent will tell you that they think they know of an instance where a colleague has committed fraud.
That blurs into the second thing, which is bias. Everyone wants to change the world, so people are biased towards finding significant results. And that can make them see things that aren’t there. They might make arbitrary changes to their statistical analysis – called p-hacking – to find the result they want. Drop a few participants here, change a number there and convince yourself that it’s the right thing to do.
The third cause is unforced errors, like a typo in a spreadsheet that knocks the whole result off track. That is negligence.
The last category is hype, where scientists are pushed towards writing up their results as if they are much more exciting than they are. The way papers are written now often makes it sound as if the findings are going to revolutionise the way we think about things. But we’re not making more groundbreaking discoveries than we used to. It’s just that there’s pressure on how we’re supposed to write up science, and lots of media excitement.
If you put all those four together, it looks as if something is very rotten.
How did we let the rot set in?
It crept in due to a system of perverse incentives. There’s huge pressure to publish papers and huge pressure to bring in grants – which is an incentive to publish papers and apply for grants, but not an incentive to discover the truth. We focus far too much on rewarding people who have brought in big grants or published papers in prestigious journals, which isn’t necessarily getting us what we want.
But there are checks and balances, like peer review, where independent experts vet papers before they are published…
It works in some cases, but it’s nowhere near the filter it needs to be. Some of the worst papers ever went through the peer-review system of the world’s best journals.
For instance, the system isn’t set up in a way that peer reviewers can easily get raw data. It’s absurd. The people who are supposed to be checking whether the analysis is correct rarely see the data that the claims are based on.
Reviewers are also often rushed. They themselves are busy scientists. Journals want there to be only a short time between articles being submitting and accepted, so peer reviewers are pushed towards accepting things really quickly. And journals only want to accept the flashiest and most exciting claims.
That’s before you get into any personal vendettas that some peer reviewers have for one another. Even if the paper is anonymous, you can usually tell who wrote it. If it’s your rival lab, maybe you’ll give it a very harsh review.
Another check in the system is replication. But that seems to have gone wrong.
I remember being told when I was an undergrad that a paper should be written in such a way that anyone could replicate the experiment. But when people try and do that, often they can’t even set the experiment up because the paper doesn’t contain enough information. When they can replicate the experiment, they often find results that are different or not statistically significant.
Scientists are not being fully open or transparent. They can often be quite reluctant to give more details when someone comes along and says, “I want to replicate your work”, because there’s this culture of, “Oh, I know best how to do this experiment.”
How much published science isn’t replicable?
In my own subject, psychology, replication was recognised as a major problem from about 2011. About 50 per cent of studies in the literature don’t seem to be replicable.
That replication crisis spread into pretty much every subject that has been looked at. In most subjects, if you ask scientists, “Do you think the research is completely robust?”, a huge proportion say no. But it’s hard to put a number on it because almost nobody else has made a systematic attempt to replicate results. My guess would be that the picture is less bad in other subjects, but we don’t know.
“Scientific papers are seen as this almost sacred thing. But a lot of people don’t know how the sausage is made”
I think that is itself a demonstration of the problem. We are not clear what proportion of papers from biomedicine or chemistry or physics and so on would be replicated because replication is not incentivised. Going back and doing someone’s experiment again, is, in most cases, not what the journal system and the university system want.
Is the replication crisis getting worse?
I think there’s evidence that things are worse now than they were 30 years ago in terms of this obsession with publication. At least in some cases, people who finish their PhDs now are expected to have some astonishingly high number of peer-reviewed publications, something like 19. A few years ago, you’d be expected to have five or six. The quality of the work inevitably has to suffer.
This a long-standing problem.
Why write a book about it in 2020?
We’ve got to the point where we have a handle on the issues. We’ve also got a great deal of so.called meta science, the science of science. People have spent their entire careers running large replication attempts. So I thought it was a good time to summarise where we are.
A lot of it is still a bit ivory tower, discussed only within science. When we’re communicating to the public, we just go back to “here’s a new exciting paper that tells us this is true” rather than revealing how the process really works.
Why should people who aren’t scientists care about this?
So much of what we do relies on science. So if the whole system is off, then that has major consequences. It couldn’t be more clearly illustrated than with covid-19. All of the things I’ve talked about have come up in the pandemic: accusations of fraud, retractions from top journals, screwed-up statistical analyses, lack of data sharing, negligence. And it has put rocket boosters under hype.
Another lesson we’ve learned is the dangers of pushing science out too quickly. We’ve seen a huge upsurge in research about covid-19, but a lot of it is probably completely useless. It’s kind of unbelievable. There couldn’t be a better illustration of the need for change.
Also, a huge amount of taxpayers’ money is wasted on useless science. It actually makes it harder to discover the truth in future because it muddies the picture.
What is at stake? If science just shrugs and carries on as normal, what will happen?
At the moment, people have a high degree of trust in science and I think that’s justified to some degree, certainly compared with politicians. But if people keep publishing these hyped results which end up not actually delivering on the promise, trust in science will decline.
We see it already in nutrition research. I think most people don’t take that seriously any more. I would hate to get into a situation where other sciences are in that same boat and research becomes a laughing stock.
Do you worry about giving ammunition to anti-science voices?
I do. I worry particularly about already politicised areas like climate change, vaccines and evolution. The last thing we want is to give climate deniers and creationists yet more reasons to be critical of science.
I’m critical of the way science is done, but I’m not critical of the idea that science is the best way we have of discovering how the world works, of making new things that improve the world for everyone. That is the whole reason that we must be sceptical and critical of it. I would be disappointed if people took this book as saying, “Well, we can just chuck away science.” That’s absolutely the opposite of what I’m trying to advocate.
How can we fix it?
I think the broad answer is scientists being more open and transparent about what they actually did in their research. But I don’t think that’s the only answer. Universities need to stop hiring people just on the basis of how many papers and citations they have and start rewarding a different kind of science: people who contribute data to the world, people who are part of large replication projects, people who are creating new tools to help other scientists do research more efficiently. Being a good scientific citizen, essentially.
Journals need to change the way they publish articles. They can incentivise people to preregister their work, so they’ve published their data analysis plan before they actually get to the analysis and can’t do stuff like p-hacking.
Ultimately, I feel like this is an optimistic book, even though it is essentially a litany of human failure and weakness.
The amazing thing is that even with all these problems, we can still know stuff and make progress. But imagine how much more we could know if we managed to build a better system than the one we have. We can do much better. We can change the way the system works.
But I also think we need to change what we think science is, and have different expectations of it. We should expect scientists to be much more open, but also more boring.
The perverse incentive to make research look exciting is at the root of a lot of the problems. Science is not an endless march of exciting, flashy findings. There are transformative discoveries, and we should try and encourage those. But in general, science is incremental and small scale and requires a new kind of intellectual humility. I think we will be much better off when we realise that.
Ritchie’s view on the need to have peer-reviewed papers to complete a PhD has been clarified
