Traditional literature review / narrative review:
Source: Cochrane. Background to Systematic Reviews
This link will open a PDF document.
A systematic review is defined as “a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” The methods used must be reproducible and transparent.
Source: Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.
Image: EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.
When answering questions of effectiveness comparing two different treatments or interventions.
Is your review question a complex intervention? Learn more about Reviews of Complex Interventions.
Choosing a Review Type: This guide explains other comprehensive literature reviews of similar methodology to the systematic review.
Here is a helpful article about review types. (Meeting the Review Family: Exploring review types and associated information retrieval requirements, 2019,Sutton et al.)
You may also find the Review Ready Reckoner helpful!
1. Gathering your team (Minimum of two reviewers with a third to serve as a tiebreaker)
A systematic review must have a team of two or greater. A systematic review cannot be completed by one person. Choose team members wisely and based on areas of expertise. A third team member is sometimes called a tiebreaker. They are to resolve disagreements for reviewers 1 and 2 for stages of the review that are blinded (screening, data extraction, critical appraisal) and are completed by two independent reviewers.
2. Questioning (Define a narrow question, may use PICO)
The PICO format is commonly used to define the research question into one that is a searchable question. In some cases, the PICO format may not work and another format can be used. The P in PICO is Patient/Problem or Person. I is for Intervention/Exposure/Therapy or Treatment. C is optional and is for Comparison (such as a placebo or another drug/therapy) and O is for Outcome(s), what is the expected or anticipated outcome you will find in the literature? A systematic review question should also be narrow in scope in order to be able to conduct the review in one year. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-defined and narrow question.
3. Planning (Create a protocol, plan methods & strategies, register protocol) * This course focuses on the planning stage
Having a plan in place is essential to a good quality review and by spending more time planning before the review takes place, you could avoid issues or errors that may slow down the process or be detrimental to the review. Planning includes seeing if the review is feasible, checking to make sure there are no conflicting reviews and also ensuring that there is a plan to carry out each stage of the review. Setting goals and timelines for the review is important as well as mapping out how the review project will be managed. This is also put into a document called a protocol. Registering the protocol is optional but highly recommended. The protocol also includes defining a priori what the selection criteria will be for the review in terms of inclusion and exclusion criteria for what studies should be screened by for inclusion in the review.
4. Searching/Screening (Exhaustive, transparent & repeatable searching for evidence/selecting studies)
Includes searching multiple databases, grey literature/clinical trial registries and handsearching of the literature (performed by the subject matter expert). It is best practice to involve a librarian or an information specialist in creating the comprehensive search, translating the search for databases or grey literature, documenting the search and deduplicating the repeating references in a citation manager and writing the search methods for the review. However, librarians are usually not involved in grey literature searching unless they are an expert in the subject matter. The review team member with the most subject matter expertise is the one who is best equipped to handsearch. The search stage may also include contacting other experts in the field to identify publications that have not been published yet. Systematic reviews include both published and unpublished literature to avoid a type of publication bias, called positive outcomes bias since positive outcomes are more likely to be published. Screening is done in two phases. The first phase is screening titles/abstracts (together) and the second phase is screening full texts. Screening is done independently by two reviewers, with a third reviewer serving as a tiebreaker. Reviewers should not move on to the full text screening phase until they have screened all of the titles and abstracts and each is a clear Yes or No without maybes remaining. Once they are ready to screen full texts, they must acquire and read all of the full texts and screen them based on the studies selection criteria. Only Yes's are included in the review but all No's must have a reason listed for exclusion. The new PRISMA 2020 requires reporting of study Near Misses too. Near misses are any studies that did not meet inclusion in the review but were very close to being included. Refer to the PRISMA 2020 http://www.prisma-statement.org/PRISMAStatement/ for more guidance on this stage. There are tools designed specifically to assist with the systematic review screening phase.
5. Managing & reporting
All methods must be fully reported, transparent and reproducible. The methods reported must also follow the recommended reporting guidance such as the PRISMA 2020. Reporting guidance can be identified by searching the Equator Network https://www.equator-network.org/. Reporting guidance may be modified for review types similar to the systematic review. Refer to the many PRISMA 2020 extensions http://www.prisma-statement.org/Extensions/ for more information.
6. Data Extraction/Synthesizing the evidence
This stage includes appraising the evidence, interpreting results, performing a qualitative (narrative analysis) and/or a quantitative/meta-analysis. A meta-analysis is optional and is only done if it is feasible. A biostatistician or advanced training in statistics is recommended if doing a meta-analysis. There are many tools designed to assist with this process.
Evidence from studies are assessed using critical appraisal or Risk of Bias tools and/or checklists by study design.
Critical appraisal tools from Temple University, Systematic Review Research Libguide
More tools for critical appraisal and other stages of the review from the National Collaborating Centre for Methods and Tools
Data from all studies must also be extracted and put into tables/charts such as the Summary of Findings (SOF) table and is reported as a narrative synthesis. Data is collected from all studies if conducting a meta-analysis and its numerical findings are reported.
Here are some more detailed elaborations and examples:
Synthesis: Provide a narrative synthesis of the included studies individually and when combined (What are the differences and the commonality between studies?) or what can be demonstrated from the research when combining the studies together? A meta-analysis is optional. Create a data abstraction/extraction form for the purposes of collecting data that is similar across all included studies, include a ‘Characteristics of Studies’ table to show this data (see table example and this example (opens a PDF document) on page 48. Summary of Findings tables are provided starting on page 8 of the same document. Data extraction must be done using data extraction forms and independently/blinded by two reviewers, with a 3rd reviewer serving as a tiebreaker.
7. Drawing Conclusions, Writing & Publishing
After completing these steps, the results of the review must be shared. What is the level of evidence? Is there evidence in support of the question or are more studies needed to draw conclusions? What are your recommendations for future studies? What are the limitations to your systematic review? How do these findings from your review change what is known on the topic or question?
Discuss what contribution this review makes and how your review answers or addresses the original question. Discuss any gaps found in the research. Make recommendations for needed research to address these gaps and the importance of addressing them. Discuss the overall strength of evidence in support of your original question (strong, moderate or weak).
For more guidance on the systematic review stages, refer to the Cochrane Handbook (medicine/health sciences), the JBI Manual (health sciences/nursing) or the methods guides for Campbell Collaboration Systematic Reviews (Business, education, social welfare, criminal & justice topics and more).