Validation, limitations, and reviewer responsibilities

Validation posture

Outputs should be interpreted as evidence-backed workflow artifacts: search candidates, screening suggestions, extracted values, appraisal drafts, model results, certainty drafts, and report drafts. Each stage is designed to be inspectable before it becomes part of the final review.

www.axelium.ai · Analysis

Automation and human review

• AI screening suggestions can be wrong or incomplete; reviewer decisions should control formal inclusion.
• Extracted values can be affected by table structure, reporting ambiguity, OCR quality, or conflicting sources.
• Risk-of-bias and certainty assessments require reviewer validation before publication.
• Generated reports are drafts and must be checked against the protocol and evidence.

Search, screening, and full text

Search coverage depends on source availability, indexing lag, query quality, and access rights. Optional source coverage and citation chasing improve recall but do not guarantee completeness. Full-text retrieval can be blocked by paywalls, publisher gates, or poor metadata; manual upload and the browser companion help resolve these cases when you have lawful access.

Screening checks its own reasoning for consistency before a record is decided automatically. If a screening rationale argues against its own eligibility answer — for example, marking the population criterion as met while explaining that the study organism falls outside it — the record is held as unsure for you to settle. The same applies when an exclusion’s stated reason is contradicted by the screening’s own assessment of the paper — for example, excluding a record as secondary literature while assessing it as a primary study report — rather than letting a confident-sounding but self-contradictory verdict drop an eligible study.

Records with no abstract are screened on what genuinely exists. A bibliographic citation is never treated as if it were an abstract, but when no abstract exists in any source — common for older literature — screening considers the title together with the publication-type labels the record is indexed under (for example, “Randomized Controlled Trial”), and marks any decision made this way as title-based so you can confirm it. A record without even a usable title is held for review unscreened.

Records excluded for their design rather than for one of your eligibility criteria — secondary research, or work screening was confident falls outside your question for another reason — are excluded rather than left in the unsure queue. Screening still holds back anything it is genuinely uncertain about, so the unsure queue reflects real ambiguity rather than decisions it could not file.

Full-text screening

The AI reviewer settles a criterion at full text only when enough independent passages it quotes are found in the paper: two by default, or one or three if the review changes it. Absence of evidence never excludes a study: a criterion the paper does not settle, or settles both ways, goes to a reviewer as Needs review. This rule reduces unsupported exclusions but does not prevent a misreading. Check the quotes behind every exclusion you rely on.

A passage can be found in the paper and still not answer the criterion — a treatment called “pre-emptive” is not a pre-emptive procedure, and a measurement taken before an operation is not the timing of the operation. Headings, table labels and passages that only relate to a criterion are therefore shown but not counted, and before a decision is confirmed automatically a separate check must agree that its quotes answer each deciding question. Both reduce, but do not remove, decisions resting on a misreading; the study view shows every quote behind a verdict, counted or not.

• The AI reviewer reads the paper's text and can miss a criterion stated only in a table or figure, which then stays unsettled or is settled on the text alone.
• When a review requires the study itself to compare the groups and chooses its full-text criteria, a passage about the comparison counts only if it states a result for the groups compared: figures, an estimate, or a stated lack of difference. A sentence that only describes the design is shown as “states no result” and not counted, so a study that reports its comparison only in a table or figure the reviewer cannot read stays unsettled and waits for you.
• Criteria a review newly requires at full text, and (with the thorough check) decisions settled without a quotation from the study, never exclude a study on their own: such a study waits for a person, and the criteria panel says why.
• Papers whose text could not be extracted, such as some scanned PDFs, and studies with no retrievable full text are marked Full text unavailable. They are not excluded for that reason and are not screened at full text until a readable copy is added.
• A reviewer's answer to a criterion settles it for that reviewer, whatever the AI reviewer found; the AI reviewer's own assessment, and the counts generated reports give of it, are unchanged. Read the quotes, or the paper, before you answer.
• Re-checking automatic decisions reads each paper again, so a record can change because the second reading differs, not only because the rules did. Read the reasons given for each record before you accept it; nothing changes until someone does.
• A document marked Not verified has not been shown to be the study's paper, which is not the same as being shown to be another's; Confirmed by a person does not say who confirmed it. Check the document the AI reviewer read before relying on its decision.
• For PRISMA 2020 item 8, generated reports state in their Methods the model and prompt version used, the evidence rule decisions were made under, how many decisions were confirmed automatically, and the exclusions by reason. Check that paragraph against your protocol before submission.

Extraction limitations

Complex tables, unusual outcome definitions, supplement-only data, multi-publication study bundles, and inconsistent registry versus publication values all require reviewer attention. Use source links, confidence indicators, and review queues to verify important values before pooling.

A study may report an effect measure that differs from the one you configured for an outcome — a hazard ratio where the outcome is set up as a risk ratio, or the reverse. Either way the reported estimate is kept and the disagreement is flagged for review, but the outcome is still pooled on the measure you configured. Where the outcome is not set up as a hazard ratio, a reviewer can record the measure the paper reports, which keeps that value out of the outcome's pool, or confirm that the value is the configured measure; where it is, the reviewer confirms, clears or re-extracts the value. Confirm the measure before relying on such a pooled result. The measure is read from the words of the quoted sentence, so a quote that names none, or names two, is not flagged.

Extraction also flags a quote whose wording suggests a subgroup, another study or a pooled result, and notes one found in the Introduction, Discussion or reference list. These checks read the sentence, so they can miss a subgroup it does not name and can flag a sentence that is the trial's own result. A flag keeps the value out of every bulk validation (Validate study, Validate all studies, the triage matrix's Validate, Import data) and Validate row until a reviewer answers it; neither a flag nor a note changes the value.

The Verify table files every extracted value under one of three words. Needs you: something speaks against it — a check, a failed automatic check, a disputed quote, an implausible effect, a missing field, a suspected swap of the arms, or a registry record that disagrees — and a person decides, one value at a time. Unconfirmed: nothing speaks against it, but its evidence is not confirmed: the quote has not yet been checked against the paper, was found only as a close match or not at all, the document holds no text to check against, no quote supports the value, or the automatic checks left no record. Ready: its quotes were found in the paper and the automatic checks passed. Bulk validation takes Ready values only; the confirmation offers the Unconfirmed ones as a choice. A quote's check is a text match against the stored document, so a value can be Ready with a wrong number quoted correctly, and Unconfirmed because a table was never parsed: the word says how the evidence was checked, not that the value is right. An outcome the extraction could not finish is listed as such, apart from one the paper does not report.

Quote highlighting and evidence excerpts depend on the PDF carrying a text layer. Scanned PDFs are shown page by page with nothing highlighted, and an excerpt is only produced once the quoted sentence has been located — by the viewer or by a reviewer. An excerpt is a picture of the stored page around the citation; it shows where a value was read from, not that the value was read correctly. Documents held only as text (registry records, full text fetched from an open-access archive) and documents whose licence terms do not permit keeping an excerpt have none.

Statistical limitations

• Small numbers of studies make heterogeneity, asymmetry, and subgroup checks unstable.
• A single multi-arm randomized trial can provide direct comparisons, but cannot estimate between-study heterogeneity or independent inconsistency. Its shared-arm comparisons are correlated, so a three-edge triangle is not three independent trials. Prediction intervals and rankings are withheld in this case; a displayed numerical zero would not establish homogeneity.
• Model choice should be justified by the protocol and data, not selected after seeing the preferred result.
• Effect measures, timepoints, and direction of benefit must be harmonised before pooling.
• Exploratory analyses should be labelled as exploratory in reports.

Custom (user-defined) analyses

• Custom R analyses run your own methods in a secure workspace against a frozen copy of the dataset. Their results are labelled “User-defined” and are not validated by Axelium's curated statistical pipeline.
• User-defined results never feed certainty-of-evidence assessments or generated reports; treat them as exploratory until independently verified.
• Reproducibility depends on the recorded package manifest; methods from third-party or unpublished packages carry that package's own correctness risks.
• Every custom run requires your explicit approval and executes with network access disabled; only package installation from public R package repositories is permitted, in a separate step.

Network meta-analysis

Axelium supports clinical network meta-analysis with reviewer-approved treatment definitions, readiness checks, and certainty review. Network setup warnings, multi-arm study caveats, sparse evidence, and inconsistency signals should prompt methodological review before results are reported.

• Each outcome is analysed on its own settings; an outcome without a measure, a direction of benefit, a reference treatment or a follow-up window does not run, and the setup page says which is missing.
• The analysis runs only after every automatically grouped arm it uses has been confirmed. A treatment group excluded from the network is left out of every outcome.
• A network in separate parts cannot compare treatments across the parts; consistency cannot be assessed without independent indirect evidence. A loop drawn entirely by the arms of one multi-arm trial is not an independent evidence loop. Both are said plainly rather than estimated around.
• When enabled for a review, the arm editor keeps one randomized trial together while recording each outcome's arm values, source evidence, structured treatment details and follow-up context. Randomized roster Ns stay distinct from outcome Ns; reviewers must confirm a denominator's population rather than silently borrow a count from another outcome.
• Results are drafts until an outcome is locked. Locking — or finalising the certainty review — fixes that outcome’s treatment groups, settings and evidence; other outcomes stay open. A run whose evidence or settings changed since is marked out of date.
• The evidence hints on the certainty page describe the numbers; they never set a rating, and a suggested overall certainty can be changed with a reason.

Ecology and evolution meta-analysis

Ecology/evolution support is in early access and currently centres on multilevel synthesis for dependent effects, with an optional additive meta-regression on recorded moderators and narrative certainty. Results are reported on the effect measure's natural scale with its null anchor.

• Phylogenetic meta-analysis is not supported.
• Moderator meta-regression is additive only; interaction terms between moderators are not fitted.
• A total heterogeneity summary with prediction intervals is reported for multilevel models; a partitioned breakdown across variance components is not.
• When sampling covariances among dependent effects are not supplied, those effects are treated as independent and the pooled uncertainty may be understated; this assumption is surfaced on the result.
• Clinical GRADE, CINeMA, and ranking-style clinical interpretation are outside the current public scope.

Qualitative and mixed-methods synthesis

CASP appraisal, coding, theme development, and CERQual confidence ratings are reviewer-owned. Automated drafts can accelerate the groundwork, but analytical findings and confidence rationales must be reviewed, edited, and finalised by the review team.

Review automation

Automated review may flag missing data, stale appraisals, unit issues, or protocol-level suggestions. Protocol-level changes should pause for explicit approval. Applied changes should be disclosed when they affect the reported review. A deep audit of extracted data can also repair narrow, well-evidenced issues automatically. Automatic repair is deliberately bounded: it never overwrites values a reviewer has validated or edited, every repair is re-checked before it counts and listed on the review page, and anything that cannot be confirmed — or that would change the protocol — comes to a reviewer. It is not a substitute for reviewer judgement about what the review should include.

Access, teams, and billing

Team roles, billing status, and connected-app authorization affect who can start work, review evidence, approve changes, and capture full text. Keep team membership and connected apps current for long-running reviews.