Gideon Meyerowitz-Katz is an epidemiologist and writer based in Sydney, Australia. His work focuses on chronic disease, pandemic response and, more recently, error detection in science. In this editorial, he discusses research issues that have become increasingly evident during the pandemic.
There are no two ways about it: science is flawed. We’re not talking about philosophical trends in science or the origins of white coats and linoleum floor labs, but the inner workings of the process by which we determine whether things are right or wrong.
In the decades leading up to the pandemic, scientists spent endless hours grappling with the painful fact that much of the knowledge base in science and medicine rests on research that is imperfect, broken, or potentially never produced. .
Science has a gap between its mechanical and the exits. The mechanics of science are good. Machines are always getting bigger and more efficient. New tools are always being developed. Techniques become more sophisticated over time, and more knowledge is acquired.
the the exits of science are not. The culture of academia demands publication and warrants little retrospection on potential errors – this means that errors are seldom corrected, and even outright fraud often goes undetected in academic literature.
And then came a pandemic, and the science gap widened into an inevitable sinkhole. While biomedical research had obvious and immediate success in mitigating COVID-19, it was accompanied by a huge tidal wave of waste, which instantly overwhelmed our waste mitigation mechanisms.
From fraud and pointless research to articles so full of errors it is astonishing they were published, the pandemic has produced a tidal wave of dismal scientific output that has nonetheless had staggering consequences. on people’s lives.
Take ivermectin. It is an incredibly effective pest control drug that has treated literally billions of people since its invention, and it has almost eliminated some parasitic diseases from the world.
It has also been promoted around the world as a cure for COVID-19 by a group of passionate fans. It is likely that more ivermectin has been taken to prevent or treat COVID-19 than any other medicine, with the possible exception of dexamethasone.
And yet, we have no idea at all whether ivermectin is actually helpful in the treatment of COVID-19.
A recent review of the Cochrane collaboration – long considered the gold standard in medical research – concluded that ivermectin should not be used for the treatment or prevention of COVID-19 outside of well-conducted clinical trials, This is in stark contrast to the hundreds of millions of doses still taken for these exact reasons.
In early 2020, people were desperately seeking any kind of treatment for COVID-19. A mixture of partial evidence has emerged.
This included: a lab study showing the drug worked as a strong antiviral in a petri dish, a study in a French nursing home where residents took ivermectin to treat a scabies outbreak and subsequently appeared benefit from higher survival rates, and a pre-print reporting that ivermectin reduced COVID-19 mortality by 90%.
All three were weak evidence in different ways. Single in vitro studies do little to predict potential clinical outcomes, and the nursing home article was an accidental and uncontrolled observational study – and if residents had never been exposed to SARS-CoV-2 in first place ?
The clinical study was completely fabricated and later removed from the preprint server, following a big scandal.
The story of ivermectin got even worse from there. At the end of 2020, studies began to appear showing what can only be described as simply amazing results for the drug – a 90% mortality benefit or a 100% reduction in cases when it is is used prophylactically.
After nearly a year, myself and other data sleuths demonstrated that many of these studies probably never happened, but the damage was well and truly done long before the first fake paper was ever made. retracted.
A meta-analysis of ivermectin, which is generally considered the gold standard of research practice, found a huge benefit for the drug. However, the document was not corrected, even though the studies underlying its results were likely to be fraudulent.
In any other discipline – media, government, private enterprise – such analysis would be immediately withdrawn with apologies. Instead, the article is allowed to reflect the general disinterest of the scientific world in correcting errors.
This story could have been told very differently. Imagine a world where the original lab document came with a disclaimer, where the fraudulent preprint was immediately viewed with skepticism, and where positive trials were assessed for fraud even before being published.
Instead, at every step of the way, the process of highlighting concerns about the data is skipped, with peer review being the only fragile obstacle to publishing terrible research.
When we needed effective fact-checking the most, our major scientific research institutions instead reviewed studies within days, if not hours, and posted fraudulent studies online to share around the world.
It’s tempting to say that ivermectin research is particularly flawed, but that’s clearly not true – realistically, it would be remarkable if a faulty system only produced one failure.
Trials of favapiravir, another reused COVID-19 drug, have recently been
There are now nearly a dozen studies looking to see if vitamin D has any benefit in COVID-19 that have been fully corrected or removed in the past 18 months.
The Retraction Watch website maintains a running tally of pandemic-related studies that have been withdrawn. At the time of publication, the figure is 199 and is increasing weekly.
Worse yet, these are just the papers that people have looked at. Scientific errors are rarely noticed because there is simply no reward for pointing out the mistakes of others.
If we were to start looking at all the unnecessary, unnecessary, and terribly done research, we could expand that number to thousands, if not tens of thousands of articles.
There are published ecological studies on ivermectin – where researchers compare drug use of entire countries and mortality from COVID-19. These studies use
A vitamin D study was pulled from the SSRN preprint server after it became clear the authors mistakenly labeled it as a randomized trial, although they did not randomize the participants at all. It has since been reposted largely unchanged, with no mention of the previous retraction in the final document.
None of this is to say that there is no good science. Vaccine trials alone are perhaps the most impressive scientific work ever, with effective immunizations developed, tested and tested in less than a year.
The problem is, large, well-conducted clinical trials are far from the norm. In a recent
Without even carefully evaluating these studies, we can say that most of them were probably a waste of time.
Indeed, if you look at the meta-analytical model of this review, virtually all of our knowledge about hydroxychloroquine for COVID-19 comes from just two studies, which recruited around 70% of all people who had previously been treated with this drug. tested.
This despite nearly 300 trials of the drug registered on clinicaltrials.gov, and the highest research spending of any drug at the start of the pandemic.
If all of these little essays had been linked together they might have gotten something useful, but instead we’re left with two good studies and a handful of largely unnecessary research.
All of this is perhaps the predictable result of a system that puts publishing first and punishes error checking with disdain, contempt, and prosecution. Publishing a bad study can earn you praise and promotion; at worst, it could end up somewhere on your resume.
Public error checking earns you a regular payment for hate mail and death threats, and it doesn’t earn you any of the quotes, posts, and awards that academia considers important.
Science has huge problems. Unless we can find a way to reward error checking with real money, we will continue to accept only a disturbing proportion of our research results – the studies we use to make life decisions or of death – is either false or incredibly problematic.
While it’s tempting to see this as a tiresome problem among egg heads, it couldn’t be further from the truth.
It is not unlikely that you or your family have been personally impacted by bad research during COVID-19 – perhaps you received hydroxychloroquine during a hospital stay or took metformin just in the morning. case where. Maybe you live in a place that has reopened schools based on a study with math errors, or you have been told that the masks constitute child abuse due to a document that has then been withdrawn.
Overall, there is a real impact of bad science in our daily lives that the pandemic has highlighted.
Worse yet, we know another pandemic is coming. If we don’t fix these problems now, the next time a new disease spreads in our world, we will be doomed to repeat the mistakes of COVID-19. And this is perhaps the most disturbing thought of all.