VU Systematic Literature Reviews: Risk of Bias

Risk of bias overview

The Cochrane Handbook defines bias as '' a systematic error, or deviation from the truth, in results" (Cochrane handbook, Ch. 7.1).

The PRISMA 2020 Statement requires reviewers to report how risk of bias assessments were carried out for a review. In planning a systematic review, reviewers need to identify the methods and tools they will use to assess:

  • The quality and risk of bias of  individual studies included in the systematic review 
  • Risk of bias due to missing results in a synthesis (Cochrane handbook, Ch. 13)
  • The level of certainty (or confidence) in the body of evidence for each outcome assessed

The resources on this page will assist reviewers in planning how to assess risk of bias of in a systematic review.

Risk of bias/ critical appraisal tools

The methods and tools used for assessing risk of bias of individual studies should be decided at the protocol stage of the review. Details of how risk of bias assessments were carried out  should be  included in the systematic review as well as stating the number of reviewers involved. 

Note that some publishers of systematic reviews may require a specific tool to be used in risk of bias assessment.

Critical appraisal tools

The tools used for assessing the quality of individual studies will be determined by the type of studies you have included in your review.

AMSTAR 2. A critical appraisal tool for assessing the quality of systematic reviews that include randomised or non-randomised studies of healthcare interventions. Checklist. Guidance Document. 

CASP Critical Assessment Skills Programme checklists. There are  eight critical appraisal tools designed for use with Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule. 

Centre for Evidence-Based Medicine  Critical appraisal worksheets from the CEBM, Oxford University. Includes worksheets to use for critical appraisal of systematic reviews, RCTs, prognosis studies, diagnostic studies and qualitative studies. 

JBI’s Critical Appraisal Tools. There are thirteen checklists designed to assist in assessing the trustworthiness, relevance and results of published papers. Checklists for critical appraisal of RCT's, systematic reviews and qualitative research are included here. 

Risk of bias tools

Cochrane risk-of-bias tool for randomized trials (RoB 2).  The recommended tool for use in Cochrane Reviews of randomized trials. 

Cochrane ROB-ME A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis

Non-randomized studies of interventions

 Cochrane Risk Of Bias In Non-Randomized Studies - of Interventions (ROBINS-I)

 Newcastle-Ottawa Scale  Tool for quality assessment of nonrandomised studies, including case-control and cohort studies.


Assessing the body of evidence

PRISMA Statement 2020

Item 15: Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.

Reviewers need to decide which factors they will include in an assessment of the certainty of the body of evidence and which tools or  framework they will use to assess the body of evidence for each outcome. The  factors considered (such as precision of the effect estimate, consistency of findings across studies) and the criteria used to assess each factor when assessing certainty in the body of evidence should be reported. It is recommended to include evidence summary tables to report assessments of the certainty of evidence. (Prisma 2020)


The GRADE approach to rating the quality of evidence begins with the study design and then addresses five reasons to possibly rate down the quality of evidence (study limitations; consistency of effect; imprecision; indirectness;  publication bias) and three to possibly rate up the quality (large magnitude of effect; dose-response gradient; all residual confounding would decrease magnitude of effect (in situations with an effect)). 

Assessing the quality of individual studies - examples using CASP

Cochrane Mental Health. (2019, March 29). Introduction to critical appraisal [Video]. YouTube.

The first in a series of seven videos that provide a step by step introduction to critical appraisal of research. In the first video the fundamental concepts of critical appraisal are explained, validity, trustworthiness of results, value and relevance. Each of the videos in the series explores a research design and applies the CASP tool in conducting a critical appraisal. Click on the links below to view the videos in this series..


GRADE and summary of findings tables: Learning modules

Mac GRADE. (2021, February 18). Choosing a comparison and outcomes: GRADE and summary of findings tables learning module [Video]. YouTube.

This series of eight introductory  slidecasts were prepared and narrated by Holger Schünemann and Nancy Santesso from the Department of Health Research Methods, Evidence, and Impact (formerly "Clinical Epidemiology and Biostatistics"), McMaster University, Hamilton, Canada. The slidecasts introduce the GRADE approach to grading the body of evidence in a systematic review and provide an example of  how to create a 'Summary of findings (SOF) table. Access the videos at Cochrane Training.  

Further resources

A series of articles on the GRADE framework. The first two articles provide an overview of the GRADE process, including  the evidence profile and summary of findings table.  The third article outlines GRADE’s approach to rating the quality of evidence.