Should AI read every paper, or filter first?

The question

Title and abstract screening is the most repetitive stage of any systematic review. A search can return hundreds or thousands of candidate records, and only a small fraction will turn out to be eligible trials. Reading every one carefully is exactly the kind of high-volume, low-yield work that AI is well suited to—but it raises an immediate design choice.

You can run screening in one stage: have the AI retrieve and read each candidate’s full text, then decide. Or you can run it in two stages: let an AI gate read only the title and abstract first, and reserve the expensive full-text read for the records that survive. The one-stage approach never misses anything for lack of detail; the two-stage approach does far less reading. Which one actually wins on cost, and does either reach the recall a published review needs? We wanted real numbers, not intuition.

What we did

We reconstructed the title/abstract screening pools of four published pairwise randomised-controlled-trial systematic reviews from the COVID-19 literature, running each one through Axelium. We treat these four reviews purely as a methods testbed—a way to compare screening strategies against a known, published set of included trials. Nothing here is a claim about the clinical efficacy of any drug.

For each review we rebuilt the candidate pool from the same kinds of public sources the original authors used—PubMed, Cochrane CENTRAL, and ClinicalTrials.gov—and applied the review’s own PICO-framed eligibility criteria following a PRISMA-style screening flow. We then ran two strategies head to head over the identical pool: single-stage screening, where Axelium’s AI reads each candidate’s full text, and two-stage screening, where Axelium’s AI title/abstract screener acts as a gate and only survivors go to full-text reading. Throughout, we measured both strategies against the trials the published reviews actually included.

Across the four corpora, eligibility prevalence spanned 1.7% to 9.2%—the kind of needle-in-a-haystack ratio that makes screening so costly in the first place, and a wide enough range to see whether a strategy’s advantage holds when relevant trials are scarce or comparatively common.

Finding 1 — the abstract gate almost always pays

The abstract gate held its own on accuracy. Measured against the trials the published reviews included, it recalled a pooled ~93% of eligible trials (95% CI 82–98), at a specificity of 71–85%—meaning it correctly cleared the large majority of irrelevant records before any full text was read.

That filtering paid off directly. In every corpus, two-stage screening came out 29–43% cheaper than reading every candidate in full—33% on the ivermectin–Cochrane pool, 29% on colchicine, 43% on vitamin D, and 29% on the larger ivermectin–BMC pool.

We expected that ranking to be fragile—surely it depends on how much more a full-text read costs than an abstract read? It didn’t. Two-stage stayed cheaper across the entire realistic range of that cost ratio: the break-even point only sat at a ratio of 1.2 to 1.45, well below where real full-text reading lands. In other words, as long as reading a full paper costs even modestly more than reading an abstract, filtering first wins—and you don’t need to know the exact numbers to make the call.

Finding 2 — the real bottleneck is access

Here is the twist. The strategy debate above quietly assumes the AI can actually get the papers. It often can’t. When we checked how many of the reviews’ included trials were openly retrievable, the answer was only ~76% (95% CI 62–86). Roughly one included trial in four could not be retrieved in full—and the unretrievable ones were disproportionately high-profile, paywalled trials, exactly the studies a review can least afford to miss.

This reframes the whole single-versus-two-stage question. It doesn’t matter how thoroughly an AI reads if it can never open the paper. Optimising the reading strategy buys efficiency; closing the access gap is what would actually buy recall. The most valuable screening time is increasingly spent securing full-text access, not choosing how to process it.

Finding 3 — the AI’s confidence doesn’t tell you what’s relevant

One more result worth flagging, because it is easy to get wrong. The screener emits a numeric confidence score alongside each decision, and it is tempting to treat that number as a relevance ranking— to triage borderline records by it, or surface the “most-confident includes” first. Don’t. In our data, the confidence score had no ability to rank relevance (AUC 0.51—essentially a coin toss).

The actual include/exclude decision, by contrast, was highly discriminating (AUC 0.90). The lesson is narrow but practical: trust what the screener decides, not how sure it says it is.

What we didn’t settle

This is a proof-of-concept research note, and the honest limitations matter as much as the headline numbers:

• Four reviews is a small sample. The cost ranking was consistent across all of them, but four corpora cannot establish how the picture generalises to other fields, designs, or search strategies.
• We evaluated a single AI screener. A different screener could shift recall, specificity, and the confidence-versus-decision result; we have not compared alternatives.
• These are COVID-era reviews, a period of unusually open and fast publication. That almost certainly overstates real-world full-text retrievability—the access bottleneck is likely worse, not better, in ordinary literatures.
• The recall figures assume a perfect full-text read of whatever is retrieved. They isolate the screening decision, so they do not account for errors introduced later when the AI actually reads and extracts from a paper.
• This work is a preprint and a proof of concept, not a validated benchmark. Treat the numbers as directional evidence, not settled fact.

Four things we took away