In the race to combat the COVID-19 pandemic, the world’s scientists have embraced a radically new method of disseminating information about their research, offering it quickly and without filters in the effort to understand and control this deadly disease.
But their new communication model is striking at the heart of scientific integrity, publicizing research that has been corrupted by speed, sloppiness and opacity. And now the academic world is being roiled by a question for which millions of lives hang in the balance: Is the public being well-served by the fast and free flow of research — or dangerously misled?
Nowhere is the question over scientific conduct louder than at Stanford University, where a trio of researchers are accused of promoting faulty analysis and “tipping the scale” on antibody studies that they say proves the virus is more widespread and less lethal than we feared, and that public health restrictions are too strict.
And now the university, which has also come under fire, is investigating the veteran professors’ research, a significant step in a world that cherishes credibility and reputation.
COVID-19 has changed, perhaps forever, the way scientists do their work. Faced with an overwhelming sense of medical urgency, the practice of using “preprint” servers — online platforms that allow scientists to share preliminary findings quickly, before formal and protracted evaluation — has soared in popularity, with bold and unvetted claims going straight to the public.
Peer review? It’s raucous and transparent, crowdsourced — via email and Twitter — by scores of commentators.
At no other time in history have so many critical insights been learned so quickly, upending the slow and insular traditions of science. According to an analysis by Aleszu Bajak and Jeff Howe of Northeastern University, more than 10,000 COVID-related papers have been published since January. By comparison, only 29 studies were published during the 2003 SARS pandemic.
But too much research lacks rigor and responsibility, experts say. Methods aren’t explained; statistical analyses aren’t transparent. Breaking from the usual protocol, testing is rushed into human study before completion of more basic work. Treatment trials are using flawed strategies. There is duplication of effort, creating waste. Half-baked claims are amplified by the media, then seized by left and right-wing activists to fuel conspiratorial narratives.
Does Vitamin D protect against COVID-19? The dozens of studies “fall into two categories — the really bad and the truly, unforgivably awful,” said Gideon Meyerowitz-Katz, an Australian epidemiologist who studies chronic disease.
How many of us will die on ventilators? A study in the Journal of the American Medical Association reported a terrifying 88% fatality rate — and then, in a little-noticed correction two days later, dropped the number to 24.5%.
Does hydroxychloroquine, endorsed by President Donald Trump, help? At least 18 clinical trials enrolling more than 75,000 patients are testing near-identical hypotheses. So far, none of them have shown any promise.
When launching tests of the new Moderna vaccine, researchers didn’t wait to see how well it prevents infection in animals before trying it in people, according to STAT. And in this week’s media blitz, based on tests involving just eight people, the company revealed very little data about the vaccine’s success.
Fastidious research standards may seem a luxury during a global pandemic that has claimed more than 335,000 lives, said McGill University biomedical ethicist Dr. Jonathan Kimmelman. But now, more than ever, it’s critical to do good work, he said.
“When we do research under pandemic conditions, it is important to maintain the same standard of rigor as we would outside a pandemic,” said Kimmelman, co-author of a May 1 article in the journal Science entitled “Against Pandemic Research Exceptionalism.”
“Any time you present research findings where the ink is not yet dry,” he said, “you need to be crystal clear that your findings are preliminary, provisional and unvetted.”
When the Stanford team — Drs. Jayanta Bhattacharya, John Ioannidis and Eran Bendavid — released the first draft of their Santa Clara County-based preprint, the news was stunning. The nation’s first study of its type, it found that the virus was astoundingly 50 to 85 times more prevalent than presumed. But that meant the death rate was far lower.
Yet the project raised eyebrows from the start.
Even before they started collecting data, the researchers openly questioned “stay at home” orders. Ioannidis wrote a provocative article arguing that if COVID-19 is less deadly, widespread restrictions “may be totally irrational.” A Wall Street Journal editorial by Bhattacharya and Bendavid was entitled “Is the Coronavirus as Deadly as They Say?” Bhattacharya revisited that theme in the Hoover Institution and Fox Nation program “Questioning Conventional Wisdom.”
When their preprint was published, its conclusions backed the trio’s policy arguments – and it was saddled with statistical problems.
It failed to describe key calculations and made at least five material mistakes, according to Will Fithian, assistant professor in UC Berkeley’s Department of Statistics. The population-weighted intervals in a table were miscalculated. The authors plugged the wrong interval into a formula. They made two math errors in executing that formula. And, misreading their test kit’s manufacturer insert, they used the wrong numbers for the antibody test’s specificity.
“They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong,” said Andrew Gelman, a professor of statistics and applied science at Columbia University.
Even as Fithian and other outside experts alerted the team to errors within two days of publication, the researchers promoted their findings in national media, from NPR to Fox News.
Without initially disclosing his role in the study, co-author Andrew Bogan published an op-ed in The Wall Street Journal asking, “If policy makers were aware from the outset that the COVID-19 death toll would be closer to that of seasonal flu … would they have risked tens of millions of jobs and livelihoods?”
Knowingly stepping into the partisan fray “injects their science blatantly into a political fight,” said Stanford’s Hank Greely, a prominent medical and scientific ethics expert. “It’s letting their research be weaponized for political ends, which guarantees it will be distorted.”
Meanwhile, the team calculated startling new infection estimates for Los Angeles County — using the same methodology and little explanation of how the numbers were calculated. Findings were initially announced only by press release. The study, released weeks later, states only that the authors used a statistical “bootstrap” technique to estimate sampling distribution, offering too little detail to help outside experts verify the accuracy of their conclusions.
“It’s not perfect, but it’s the best science can do,” Ioannidis told The New York Times on April 21.
Their Santa Clara County-based study was revised and republished on April 30, addressing many problems and estimating a new infection rate on the low end of the original range — but outside experts say it’s still flawed. They assert that there remains a serious methodological problem that, if corrected, would widen the “confidence interval,” or the range of possible infection rates.
Since then, questions have been raised about the project’s funding. Who is Bogan, a Palo Alto-based manager of global equity funds who left science two decades ago? . There are also suspicions about the role of David Neeleman, the JetBlue founder who is outspoken about lifting restrictions and said he “consulted” with the team. He also contributed $5,000 to Stanford for the researchers, according to BuzzFeed News.
Even more incriminating is an anonymous whistleblower complaint to Stanford, obtained by BuzzFeed, which asserts that microbiologist Taia Wang, who performed the team’s assays, refused to be an author and wrote that she didn’t trust their test. Respected Stanford pathologist Scott Boyd also tested their samples and then distanced himself from the work.
“In all studies — and especially those that could impact public health — rigorous design, statistical correctness and responsible reporting of findings are essential,” said Robert Tibshirani, a professor in Stanford’s Departments of Statistics and Biomedical Data Science. “And transparency is a key component throughout: Authors should provide full details of what they did, their computer code, and the source data, except when protected by privacy rules.”
Stanford’s reputation will pay a price for publicizing this work, “because people will remember that ‘the Stanford study’ was hyped but it had issues,” predicted Columbia’s Gelman. “The next study out of Stanford will have a little less of that credibility bank to borrow from.”
Stanford says it is reviewing the team’s work, insisting “the integrity of Stanford Medicine’s research is core to our mission. When we receive concerns such as this, they are taken extremely seriously,” said Stanford Medicine spokesperson Julie Greicius.
The team staunchly defends its research, saying the study was done in accordance with good research practice and university policies and approval. They revised their preprint with alacrity, addressing the criticisms, they add.
“I understand that the university is investigating some of the allegations that have been made in the media and expect that the study will be fully vindicated,” said Bhattacharya, senior author of the study.
According co-author Bendavid, an assistant professor of medicine, “there has been no finding of wrongdoing by the university as a result of this study, which I believe was done in accordance with good research practices, university policies, and university approval. I am confident that this will be shown to be true and that my strong academic and professional reputation will be upheld.”
Ioannidis expressed gratitude for the public critiques, saying “this is exactly how science should work. … Preprints are not immutable final versions.”
Political bias played no role in the analysis, he said. “I have repeatedly stated that science should not mix with ideology, and I find the allegation of conservative ideology for myself to be very weird given my entire career,” he said. “We are facing a very serious problem with many human lives being at stake,” he said. “My only goal is to do the best possible science and to save lives.”
Mistakes and wrong turns are endemic to the scientific process, even when scientists try their best to be careful, said UC Berkeley’s Fithian. “We’re all racing against time.”
“But I think science can and must do better,” he said. “If you learn you have made mistakes, acknowledge them publicly, before your next media appearance.”