The Cochrane Handbook defines bias as '' a systematic error, or deviation from the truth, in results" (Cochrane handbook, Ch. 7.1).
The PRISMA 2020 Statement requires reviewers to report how risk of bias assessments were carried out for a review. In planning a systematic review, reviewers need to identify the methods and tools they will use to assess:
The resources on this page will assist reviewers in planning how to assess risk of bias of in a systematic review.
The methods and tools used for assessing risk of bias of individual studies should be decided at the protocol stage of the review. Details of how risk of bias assessments were carried out should be included in the systematic review as well as stating the number of reviewers involved.
Note that some publishers of systematic reviews may require a specific tool to be used in risk of bias assessment.
Critical appraisal tools
The tools used for assessing the quality of individual studies will be determined by the type of studies you have included in your review.
AMSTAR 2. A critical appraisal tool for assessing the quality of systematic reviews that include randomised or non-randomised studies of healthcare interventions. Checklist. Guidance Document.
CASP Critical Assessment Skills Programme checklists. There are eight critical appraisal tools designed for use with Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
Centre for Evidence-Based Medicine Critical appraisal worksheets from the CEBM, Oxford University. Includes worksheets to use for critical appraisal of systematic reviews, RCTs, prognosis studies, diagnostic studies and qualitative studies.
JBI’s Critical Appraisal Tools. There are thirteen checklists designed to assist in assessing the trustworthiness, relevance and results of published papers. Checklists for critical appraisal of RCT's, systematic reviews and qualitative research are included here.
Risk of bias tools
Cochrane risk-of-bias tool for randomized trials (RoB 2). The recommended tool for use in Cochrane Reviews of randomized trials.
Cochrane ROB-ME A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis
Non-randomized studies of interventions
Cochrane Risk Of Bias In Non-Randomized Studies - of Interventions (ROBINS-I)
Newcastle-Ottawa Scale Tool for quality assessment of nonrandomised studies, including case-control and cohort studies.
Item 15: Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.
Reviewers need to decide which factors they will include in an assessment of the certainty of the body of evidence and which tools or framework they will use to assess the body of evidence for each outcome. The factors considered (such as precision of the effect estimate, consistency of findings across studies) and the criteria used to assess each factor when assessing certainty in the body of evidence should be reported. It is recommended to include evidence summary tables to report assessments of the certainty of evidence. (Prisma 2020)
GRADE
The GRADE approach to rating the quality of evidence begins with the study design and then addresses five reasons to possibly rate down the quality of evidence (study limitations; consistency of effect; imprecision; indirectness; publication bias) and three to possibly rate up the quality (large magnitude of effect; dose-response gradient; all residual confounding would decrease magnitude of effect (in situations with an effect)).
Cochrane Mental Health. (2019, March 29). Introduction to critical appraisal [Video]. YouTube. https://www.youtube.com/watch?v=H8Y-yfi3vp4
The first in a series of seven videos that provide a step by step introduction to critical appraisal of research. In the first video the fundamental concepts of critical appraisal are explained, validity, trustworthiness of results, value and relevance. Each of the videos in the series explores a research design and applies the CASP tool in conducting a critical appraisal. Click on the links below to view the videos in this series..
Mac GRADE. (2021, February 18). Choosing a comparison and outcomes: GRADE and summary of findings tables learning module [Video]. YouTube. https://www.youtube.com/watch?v=UfP3ESQ5Q3s
This series of eight introductory slidecasts were prepared and narrated by Holger Schünemann and Nancy Santesso from the Department of Health Research Methods, Evidence, and Impact (formerly "Clinical Epidemiology and Biostatistics"), McMaster University, Hamilton, Canada. The slidecasts introduce the GRADE approach to grading the body of evidence in a systematic review and provide an example of how to create a 'Summary of findings (SOF) table. Access the videos at Cochrane Training.
A series of articles on the GRADE framework. The first two articles provide an overview of the GRADE process, including the evidence profile and summary of findings table. The third article outlines GRADE’s approach to rating the quality of evidence.