Skip to main content

Systematic Reviews

Library workshop: So You Want to Do a Systematic Review

Steps and Standards of a Systematic Review

A systematic review begins with a clearly defined question accompanied by established inclusion and exclusion criteria.

Once you have an initial idea, search for already published literature on the topic. If there is already a systematic review article, then another review on the same topic is not needed. Additionally, reviewing published systematic reviews can help you to better frame your own research question.

If you find a research article that matches your topic, save it. Articles that exemplify your research topic are called "gold-standard" articles, and are used as examples during the search process (Step 3). 

Databases for Finding Already Published Reviews

  • PubMed Clinical Query (Free): A search  strategy to retrieve citations identified as systematic reviews or meta-analyses.
  • TRIP Database (Free): A tool to find systematic reviews using PICO (Population, Intervention, Comparison, and Outcome)
  • Cochrane Systematic Reviews (NetID required): The resource includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.

What is a systematic review protocol?

Based on Cochrane, it is a pre-defined plan and the proposed approach for a systematic review. It outlines the question that the review authors are addressing, detailing the criteria against which studies will be assessed for inclusion in the review, and describing how the authors will manage the review process. The protocol also serves as a notification of your plans to other researchers, so that no one will attempt the same project. It's strongly suggested to register your systematic review protocol once it's written. The purpose of registering a systematic review is to reduce publication bias, enhance transparency, and avoid duplication of effort.

A good way to familiarize yourself with writing systematic review research protocols is to take a look at those registered on PROSPERO- the International Prospective Register of Systematic Reviews. Once you have written a draft of your protocol, have it peer reviewed by someone outside of your research team. If it is registered in PROSPERO, it will also be open for peer review by other researchers.

Key information for writing a systematic review protocol include:

  • A research question including patients and population, intervention or exposure, comparison (if relevant), and outcomes.
  • Databases used for the searches and search strategy 
  • Criteria for inclusion and exclusion.
  • Methods used to assess risk of bias.
  • Method of data extraction (selection and coding form). 
  • Method of data analysis including statistical analysis if relevant.
  • Anticipated or actual start date.
  • Forming a systematic review team and team members' information.
  • Review project timeline.
  • Funding sources/sponsors
  • Conflict of interests.
  • Date of registration. 

Key messages

  • The systematic search is the most important piece of the process because it is how you collect all the data.
  • The overall goal is to locate every article that exists about the research topic so nothing is missed or overlooked. 
  • Systematic searches must be comprehensive and exhaustive in order to reduce the risk of bias.
  • Best practices include searching at least three different databases, as well as grey literature, which includes conference abstracts or papers, hard to find studies, reports, dissertations, governmental or private sector research, ongoing or unpublished clinical trials, etc.You can learn more about databases and grey literature sources here
  • Report search strategies with transparency.
  • A well-constructed systematic literature search can result in hundreds or thousands of articles to review. Please be prepared for the time commitment that is associated with a systematic review. 

Search/Methodology Filters

Methodology filters can help get rid of undesirable study designs so as to reduce the size of a large retrieval. However systematic review attempts to maximize sensitivity in terms of retrieving all relevant documents. Therefore it is always preferable not to employ filters.

Sometimes retrieval sets are literally unmanageable. You‘d probably want to implement a methodology filter. If so, use at all possible validated filters, which have been tested against gold-standard sets of bibliographic records. 

After the search is completed, the research team will review the abstracts based on specific inclusion/exclusion criteria independently. Once all the abstracts have been reviewed by each team member, the research team should meet to resolve any conflicts.

Once any conflicts from the abstract review are resolved, the abstracts voted for inclusion will move to the full-text review. The research team should gather the full-text articles and follow the same independent review process with inclusion/exclusion criteria. 

You might consider using systematic review software or other data management tools to assist in this step and the full-text review step. 

Data extraction and analysis can be done in a few different ways. You could develop a data extraction form that the research team will use. You could also use an existing software or tool to complete this step.

Data Extraction Slides from AHRQ

The slides describe why data extract is important, data extraction challenges, forms, methods to collect data accurately and effectively, etc.

You can learn more about data extraction and management here.

At the end, it is important to disseminate the information gathered by the research team. The format of the written report might depend on your targeted journal(s) for publication. Refer to their author and submission guidelines. 

Generally, the written report includes summaries and descriptions for each of the steps. 

Examples of systematic review articles:

Buchwald, H., Avidor, Y., Braunwald, E., Jensen, M. D., Pories, W., Fahrbach, K., & Schoelles, K. (2004). Bariatric surgery: A systematic review and meta-analysis. JAMA, 292(14), 1724-1737. doi:10.1001/jama.292.14.1724. PMID: 15479938

Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., . . . Shekelle, P. G. (2006). Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144(10), 742-752. PMID: 16702590

 

Do You Have Time to Commit to a Systematic Review?

Due to the methodology and standards for systematic reviews, this type of research is a significant time commitment. 

The Cochrane Handbook for Systematic Reviews of Interventions estimates a 12-month timeframe for completion. 

Month Activity
1-2 Preparation of Protocol
3-8 Search for Published and Unpublished Studies
2-3 Pilot Test of Eligibility Criteria
3-8 Inclusion Assessments
3 Pilot Test of 'Risk of Bias' Assessment
3-10 Validity Assessments
3 Pilot Test of Data Collection
3-10 Data Collection
3-10 Data Entry
5-11 Follow-up of Missing Information
8-10 Analysis
1-11 Preparation of Written Report
12- Keeping the Review Up-to-date

 

Adapted from Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from https://training.cochrane.org/handbook

Systematic Review Databases & Resources

General Biomedical Databases

  • PubMed/Medline: The premier biomedical database of the US National Library of Medicine
  • Scopus: A Multi disciplinary citation database

Subject Specific Databases

Databases for Finding Reviews

  • PubMed Clinical Query: A search  strategy to retrieve citations identified as systematic reviews or meta-analyses.
  • TRIP Database: A tool to find systematic reviews using PICO (Population, Intervention, Comparison, and Outcome)
  • Cochrane Systematic Reviews: The resource includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.

Grey Literature

Grey literature refers to "information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing" ie. where publishing is not the primary activity of the producing body." — ICGL Luxembourg definition, 1997. Expanded in New York, 2004. They include conference abstracts or papers, hard to find studies, reports, or dissertations, governmental or private sector research, ongoing or unpublished clinical trials, etc.

  • ClinicalTrials.gov: US government registry and results database of publicly and privately supported clinical studies of human participants conducted around the world.
  • UK Clinical Trial Gateway (UKCTG): Provides information about clinical research trials running in the UK.
  • The WHO ICTRP: A central database that contains the trial registration data sets provided by clinical trials registers to the World Health Organization.
  • NIH RePORTer: A searchable database of federally funded biomedical research projects conducted at universities, hospitals, and other research institutions. It also provides access to publications and patents resulting from NIH funding.
  • OpenGrey: covers Science, Technology, Biomedical Science, Economics, Social Science and Humanities.
  • Grey Literature Report: Grey literature publications in health services research and selected public health topics. 
  • MedNar:  a search tool to search websites such as medical societies, NIH resources, and other government resources.
  • Proquest Dissertation Database: A collection of theses and dissertations by UTHSC students. 
  • Grey literature producing organizations: A list of grey literature publishers compiled by the New York Academy of Medical Library 

Back to Top 

Systematic Review Types

Type Aim Question Format Question Example
Effectiveness To evaluate the effectiveness of a certain treatment/practice in terms of its impact on outcomes Population, Intervention, Comparator/s, Outcomes
(PICO)
What is the effectiveness of exercise for treating depression in adults compared to no treatment or a comparison treatment?
Experiential To investigate the experience or meaningfulness of a particular phenomenon Population, Phenomena of Interest, Context (PICo) What is the experience of undergoing high technology medical imaging (such as Magnetic Resonance Imaging) in adult patients in high income countries?
Costs/Economic Evaluation To determine the costs associated with a particular approach/treatment strategy, particularly in terms of cost effectiveness or benefit Population, Intervention, Comparator/s, Outcomes,
Context (PICOC)
What is the cost effectiveness of self-monitoring of blood glucose in type 2 diabetes mellitus in high income countries? 
Prevalence and/or Incidence To determine the prevalence and/or incidence of a certain condition Condition, Context,
Population (CoCoPop) 
What is the prevalence/incidence of claustrophobia and claustrophobic reactions in adult patients undergoing MRI?
Diagnostic Test Accuracy To determine how well a diagnostic test works in terms of its sensitivity and specificity for a particular diagnosis Population, Index Test,
Reference Test, Diagnosis of Interest (PIRD)
What is the diagnostic test accuracy of nutritional tools (such as the Malnutrition Screening Tool) compared to the Patient Generated Subjective Global Assessment amongst patients with colorectal cancer to identify undernutrition?
Etiology and/or Risk To determine the association between particular exposures/risk factors and outcomes Population, Exposure,
Outcome (PEO)
Are adults exposed to radon at risk for developing lung cancer?
Expert opinion/policy To review and synthesize current expert opinion, text or policy on a certain phenomena Population, Intervention or Phenomena of Interest,
Context (PICo)
What are the policy strategies to reduce maternal mortality in pregnant and birthing women in Cambodia, Thailand, Malaysia and Sri Lanka?
Psychometric To evaluate the psychometric properties of a certain test, normally to determine how the reliability and validity of a particular test or assessment Construct of interest or the name of the measurement instrument(s), Population,
Type of measurement instrument, Measurement properties 
What is the reliability, validity, responsiveness and interpretability of methods (manual muscle testing, isokinetic dynamometry, hand held dynamometry) to assess muscle strength in adults?
Prognostic To determine the overall prognosis for a condition, the link between specific prognostic factors and an outcome and/or prognostic/prediction models and prognostic tests Population, Prognostic Factors (or models of interest), Outcome
(PFO)
In adults with low back pain, what is the association between individual recovery expectations and disability outcomes?
Methodology To examine and investigate current research methods and potentially their impact on research quality Types of Studies, Types of Data, Types of Methods, Outcomes (SDMO) What is the effect of masked (blind) peer review for quantitative studies in terms of the study quality as reported in published reports?

Munn, Z., et al. (2018). "What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences." BMC Medical Research Methodology 18(1): 5. doi: 10.1186/s12874-017-0468-4