Findings from clinical trials and other studies are summarized in systematic reviews, which guide policy decisions.Credit: Philippe Lopez/AFP via Getty
Scientists say they have used artificial intelligence (AI) to reproduce 12 systematic reviews in two days — slashing the time it takes to produce these ‘gold-standard’ studies, which bring together the results of multiple publications and normally take many months to do.
The new system, described in a preprint posted on medRxiv last week1, could help to accelerate the production of influential reviews used by doctors and policymakers, its developers say. Using large language models (LLMs) to accelerate two laborious steps in the systematic-review process allowed the team to rapidly reproduce a dozen Cochrane reviews — a particularly rigorous type of study — and even to update a review in just 20 minutes, says co-author Christian Cao, a medical student at the University of Toronto, Canada.
Can AI review the scientific literature — and figure out what it all means?
“It’s crazy to see that in the time I get a coffee, a systematic review would be done,” says Cao. He and two of his collaborators have started a spin-off company called otto review to turn their methods into a commercial software platform.
But other researchers are sceptical that the technology can accelerate the process as much as its creators suggest. They point out that the team has not yet automated some major tasks, and that the system needs to be tested independently — which would require more details of the study to be released. “For a scientific paper, its claims need to be verifiable — and I can’t verify it,” says James Thomas, a systematic-review specialist at University College London.
Lengthy task
Systematic reviews are important because they can make sense of many academic studies such as clinical trials by combining their results. They underlie the guidelines used in medical practice worldwide and are increasingly used to guide decisions in policy, education and elsewhere.
But doing a systematic review is a chore. It involves searching academic databases to find potentially thousands of relevant studies, screening them to pull out the handful of most pertinent ones, assessing if they could be biased or untrustworthy, and extracting and analysing the data, often using a statistical technique called a metanalysis. The process takes months — sometimes years — and costs well over $100,000 per review, according to one estimate, which is why many systematic reviews are not kept up to date.
Scientists are building giant ‘evidence banks’ to create policies that actually work
Researchers have long used software to help, but the last few years has seen a surge of interest in using LLMs to automate parts of the task. “This field is awash with hypey claims from companies,” says Justin Clark, who builds automated review tools at Bond University in Gold Coast, Australia.
