Risk-of-bias appraisal

Instrument catalogue

• Cochrane RoB 2.0 (2016). For individually randomised trials, assessed per outcome. Five domains — the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result — each judged low risk, some concerns, or high risk via the tool’s published algorithm over its signalling questions, with an overall judgement derived the same way.
• Cochrane risk-of-bias tool (2011, “RoB 1”). The original domain-based Cochrane tool: sequence generation, allocation concealment, blinding, incomplete outcome data, selective reporting, and other bias, each judged low, unclear, or high risk. The tool itself mandates no overall study-level judgement, so none is invented; reviews that published one can carry it through when their assessments are imported. An optional outcome-split form judges the blinding and incomplete-data entries separately for each outcome — the Handbook permits this split, and reviews commonly use it to separate subjective from objective outcomes.
• Jadad scale (1996). A 0–5 quality score for randomised trials built from randomisation (description and appropriateness), blinding (description and appropriateness), and the accounting of withdrawals and dropouts. Scores of 3 or more are conventionally treated as higher quality. The scale has no middle category — which matters when interpreting agreement statistics against three-level tools (see below).
• Downs & Black checklist (1998). A 27-item checklist for randomised and non-randomised studies covering reporting, external validity, internal validity (bias and confounding), and statistical power. The original form has a 32-point maximum; the widely used modified form simplifies the power item to a single point for a 28-point maximum. Totals are grouped into excellent, good, fair, and poor bands.
• ROBINS-I (2016). For non-randomised studies of interventions, judged against a pre-specified target trial and confounder list. Seven domains — confounding, selection of participants, classification of interventions, deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result — on a five-level scale: low, moderate, serious, critical, and no information.

Choosing instruments

Appraisal is off until you turn it on. A review that has never been asked for a risk-of-bias assessment does not get one silently, so adding this step to an existing review is always a deliberate choice you make on the configuration page.

Which instruments appraise your studies is set on the analysis configuration page, and the configuration assistant now proposes a checklist suited to the study designs your question will retrieve — non-randomised tools for an observational corpus, RoB 2.0 for randomised trials, or both for a mixed one — naming the exact form of each so nothing is left to a default. By default each study is matched to the instruments suited to its design — randomised trials to the randomised-trial tools, non-randomised studies to the non-randomised ones — or you can have every configured instrument attempt every study, which is what a concordance comparison between tools needs. Instruments that require pre-specified review decisions, such as ROBINS-I’s target trial and confounder list, are filled in there too, and are checked before an appraisal can start. Where a tool circulates in more than one form, the exact form is selectable; forms whose transcription has not yet been confirmed against the published source are marked provisional, and you can require confirmed forms only so a batch appraisal refuses to start rather than use one. Results are then read against the form each study was actually appraised with: a shortened form is scored out of its own maximum and banded by its own cut-offs, and if an analysis holds two forms of the same tool each gets its own results view rather than sharing one scale.

The Risk of bias page always shows the current setup before you generate anything: each chosen instrument appears with the number of included studies it would assess, so a mismatch — an instrument that would assess zero studies, say — is visible before an appraisal starts rather than after. If no instrument has been chosen yet, the page says so, and for a corpus of mostly non-randomised studies it suggests a suitable tool. An “Edit instruments” link takes you straight to the selection card on the configuration page.

AI-proposed, human-validated, and imported assessments

Every assessment carries its provenance, and the three kinds are held to different standards:

• AI-proposed. The platform reads each study’s full text and proposes answers to the instrument’s own questions, with supporting quotes. Where a tool publishes a scoring algorithm (RoB 2.0) or an arithmetic rubric (Jadad, Downs & Black), the judgement or score is computed from those answers by the published rules. Where a tool publishes no algorithm — RoB 1 and ROBINS-I judgements are the assessor’s own — the proposal is clearly marked as AI-judged.
• Human-validated. A reviewer has confirmed or corrected the proposal. Proposals require human validation before they can influence certainty ratings, and only validated assessments participate as a rater in concordance comparisons. An assessment reads Draft while it is the AI’s proposal (or one the AI assessed again after its data changed), In conflict while a reviewer disagrees with it, and Finalised once a review agrees with it or a lead reviewer’s ruling keeps one side; there is no Finalise button. The Review queue is where you give your own judgement, without seeing the proposal; a review that agrees with the proposal finalises it, whether the review has one reviewer or two. One that disagrees sends the assessment to Conflicts: with two reviewers another lead reviewer rules on it, and with one reviewer you rule on it yourself, giving your reason. The foot of its panel says which of these finalises it, links the Review queue or Conflicts, names the Analysis freeze milestone it counts towards, and, once finalised, who finalised it and when.
• Imported. Assessments transcribed from a published review, attributed to a reviewer label rather than a platform user. They are treated as the published human judgement — already validated — and form their own rater in concordance comparisons.

Resolving disagreements

When a reviewer’s judgement differs from the AI proposal on any domain, the assessment becomes a conflict for a lead reviewer, who compares the two domain by domain and chooses one resolution:

• Keep the AI assessment. The proposal stands and is finalised. Resolve asks you to confirm first.
• Keep the reviewer’s assessment. The reviewer’s judgements replace the proposal’s, and the score or overall judgement is recomputed from them by the tool’s published rules, and the assessment is finalised. Resolve asks why, and the reason is kept in the record.
• Set aside. The assessment is left out of certainty ratings and reads Set aside. It can’t be reviewed or re-run afterwards. Resolve asks why, and the reason is kept in the record.
• Decide later. Nothing is decided: the conflict stays open for another ruling.

A lead reviewer can’t resolve a conflict they took part in. Viewers can read risk-of-bias assessments but can’t review, run, or import them.

Interpreting concordance between raters

When at least two raters — different instruments, or platform and published assessments of the same instrument — have rated the same studies, the risk-of-bias page shows a concordance comparison: percent exact agreement and Cohen’s kappa (with a 95% confidence interval) for every rater pair, plus a study-by-rater matrix. Read it with these caveats:

• A shared comparison scale. Each instrument’s results are translated onto a common four-level scale (low / intermediate / high / unclear) so unlike tools can be compared at all. These mappings are reporting conveniences, not equivalences — always consult each instrument’s raw results before drawing conclusions.
• Worst-of-outcomes collapse. Instruments assessed per outcome are collapsed to one study-level rating by taking the worst rating across the study’s outcomes. This is conservative by construction: a study rated low risk on most outcomes but high on one counts as high.
• Kappa’s known caveats. Cohen’s kappa corrects percent agreement for chance, but is unstable at small study counts and can be paradoxically low when most studies share one rating (high agreement, low kappa). Report both statistics, not kappa alone.
• 2×2 comparisons. When either tool in a pair has no middle category (Jadad), agreement is effectively over a two-by-two table and the pair is flagged accordingly — its kappa is not directly comparable to kappa over three or more levels.
• One model is not two raters. Platform assessments of different instruments share a single underlying assessment process, so cross-instrument agreement within the platform is not agreement between independent raters. The methodologically cleaner comparison is platform versus published assessments of the same instrument.

Importing published assessments

To compare against a published review’s own appraisal, import its assessments from a spreadsheet. Each row identifies the study (by PMID, DOI, or exact title), names the appraisal tool (and, where relevant, which form of it), and gives the value for one of the tool’s questions or domains — a judgement level for judgement-based tools, or the item’s points for score-based tools. Downloadable per-instrument templates show the expected layout, and totals and overall judgements are recomputed from the per-item values by each tool’s published rules rather than trusted from the transcription.

Imports are attributed to a reviewer label naming the original review, and each assessment drawer shows which review an imported assessment came from. Imported assessments count as validated, so importing is available to lead reviewers, and the lead who imports them is recorded as having checked the transcription. Once the analysis is frozen, imports are refused, as other risk-of-bias changes are.

Re-importing updates earlier imports in place. When a study already has an assessment with the same tool — one the AI proposed or a reviewer has worked on — the import keeps that assessment and lists the study as not imported, so nothing already in the analysis is replaced. Study identifiers that match nothing in the analysis are reported back rather than silently skipped.