Introduction to Conducting Systematic Reviews

Welcome to the Introduction to Conducting Systematic Reviews

from Health Sciences and Human Services Library.

There are entire books written about systematic review theories and methods,

and you could take a semester long course on the topic.

So we can only scratch the surface in an hour-long workshop.

This will provide you with some answers,

a clearer understanding of the systematic review process,

and perhaps motivate you to learn more.

Today, we will discuss the steps in the systematic review process,

the standards for conducting systematic reviews,

and some of the resources available

for managing your review.

Let's begin by defining systematic reviews.

Systematic reviews are often confused with

traditional reviews,

which may be called literature reviews, narrative reviews,

critical reviews or commentaries.

Traditional reviews can be useful for background information on a topic.

However, their methods differ from systematic reviews

in that it is not often possible to replicate their findings.

Traditional reviews are not as rigorous or unbiased as systematic reviews

For instance, traditional reviews are rarely explicit about how studies are selected,

assessed, or integrated.

In this type of study you can pick and choose the studies you highlight,

telling the narrative that best fits your interests.

The purpose of a systematic review is to collate existing research

on a specific question by synthesizing the results of several studies.

They should be conducted with the same rigor that is expected

from primary research.

A systematic review uses transparent procedures

to find, evaluate, and synthesize the results of relevant research.

Procedures are explicitly defined in advance,

in order to ensure that the exercise is transparent,

and can be replicated.

Studies included in a review are screened for quality by a team of researchers,

in order to minimize bias.

The overarching aim of systematic reviews should be to summarize

and help people to understand the evidence.

A number of factors may motivate authors

to undertake a systematic review.

For example,

reviews can be conducted in an effort to resolve

conflicting evidence, to explore variations in practice,

to confirm the appropriateness of current practice,

or to highlight a need for future research.

They may be used as a starting point

for developing clinical practice guidelines.

Granting agencies may require a systematic review

to ensure there's justification for further research.

Systematic reviews may demonstrate where knowledge is lacking,

and can then be used to guide future research.

Any individual research study may be flawed,

either by chance or because of how it was designed and conducted or reported.

By synthesizing the results of multiple studies,

a systematic review provides a more comprehensive picture

than a single study.

When conducted well systematic reviews should give the

best possible estimate of any true effect.

As with all research though, the value of a systematic review

depends on what was done what was found

and the clarity of reporting.

There are a number of challenges related to conducting

a systematic review, as well.

Because they're often the preliminary step in a larger project,

systematic  reviews are often unfunded and therefore can be difficult to commit to.

There is also a lot of training that must take place

before conducting a systematic review.

That includes education you need as a new PI,

but also time to train your team on the methods and tools

you'll be using during the review.

There is also the methodological challenge

of synthesizing data from multiple study designs.

Many research questions will lend themselves to analyzing more than just

randomized controlled trials,

and deciding how to treat these studies will take additional work.

Finally, even with a well-trained well-prepared team systematic reviews

are a long-term, time-consuming project.

Before you start your review, you'll want to ask yourself a few questions.

First, do you have a very clear specific topic.

If your topic is too broad or unfocused,

the review can take much longer or can end up not being viable.

More specific well-thought-out questions

generally result in a much easier process and a better quality product.

Second, is your review needed?

Determine whether a systematic review already exists on your topic.

If one does, appraise the review for its quality and recency

to decide whether it merits updating.

Finally, systematic reviews require a lot of work

and a lot of time.

Make sure you have the time to commit to undertaking a systematic review,

and make sure you have a team to help.

The time needed to conduct a systematic review

will depend on your topic and also your schedule.

Your timeline will look different depending on how much time

you can devote to a systematic review

relative to your clinical work and other responsibilities.

Cochrane estimates their reviews take about a year to conduct.

Even though you may not be conducting a Cochrane review,

six to twelve months is a reasonable estimate,

and should give you an idea of the time you may need

to devote to a systematic review.

A timetable with clear milestones will help you monitor progress,

and keep you on track.

Once you've committed to conducting a systematic  review,

you'll need to put together a team for the project.

You'll need at least two people to work on a systematic review

so that selection and data extraction can be performed

by two independent reviewers,

reducing the potential for bias and unintentional error.

Beyond that you can determine the right size for your group.

Larger groups can share the workload more easily,

but may encounter logistical problems when working together.

I've worked with groups as small as two,

but ranging to about a dozen.

To determine the right size for your group,

consider the role of each team member.

Teams should include expertise in the topic area being reviewed,

and include or have access to expertise and systematic review methodology.

This often means including a statistician

if you're conducting a meta-analysis,

or a librarian to perform the search

If you're a first time review author,

it's recommended that you have a mentor in systematic review methodology

who can be available throughout the process.

There are several resources outlining standards for

conducting systematic reviews.

The Cochrane Handbook is the official document

that describes in detail the process of preparing

and maintaining Cochrane systematic reviews.

This is a really useful document

even if you aren't conducting a Cochrane review,

and I recommend it as a resource throughout the process.

The Institute of Medicine has also published recommendations

for systematic reviews.

These are standards for systematic reviews of the comparative effectiveness

of medical or surgical interventions.

Finally, PRISMA stands for preferred reporting items

for systematic reviews and meta-analyses.

The PRISMA statement consists of a checklist and flow diagram to help authors

conduct and report the results of systematic reviews and meta-analyses.

PRISMA has been endorsed by organizations such as

Cochrane, The Center for Reviews and Dissemination, The Council of Science Editors,

and countless journals.

Remember, the value of a systematic review depends on what was done,

what was found and the clarity of reporting.

PRISMA aims to improve the quality of systematic reviews.

Now that we've discussed the basics let's talk about the process of conducting

systematic reviews.

In this portion of the presentation we'll walk through each step

of the systematic review process, from clarifying your question, to writing the report.

A clearly defined focused review begins with a well framed  question.

Questions may be narrow such as looking at the effects of treatments

or broad like the effectiveness of programs.

Systematic reviews are not limited to

questions about the effects of interventions.

They can address trends or the accuracy of diagnostic and prognostic tests.

Systematic reviews are also not limited to synthesizing randomized controlled trials,

or even quantitative data.

Some reviews synthesize studies from a variety of methods or study designs.

When beginning make sure you have a clearly stated objective.

What question are you trying to answer?

The review question should specify the population or type of patient,

types of interventions, and comparisons,

and the types of outcomes that are of interest.

The acronym PICOS helps to serve as a reminder.

These components of the question,

with the additional specification of types of study that will be included,

form the basis of the pre-specified eligibility criteria for the review.

Explicit criteria based on the reviews scope and question

are used to include and exclude studies.

Clear criteria and records of which studies meet these criteria

make it known to readers of  the review what research is informing the conclusions.

By appraising each study against the same criteria and recording the results,

the basis for the reviews conclusions is made transparent.

In non-systematic reviews it's often hard to know what evidence

the authors are drawing on and what they have discarded and why.

Eligibility criteria also help to avoid hidden bias

by having consistent rules about which studies are being used

to answer the review's specific research questions.

Each study needs to be compared against the eligibility criteria.

To be included in the review a study needs to meet all inclusion criteria

and not meet any exclusion criteria.

Excluded studies might have very useful contributions to make elsewhere

even though they're not considered relevant to the current review.

Ideally criteria should be detailed prior to starting the review.

Specifying the methods in advance reduces the risk of introducing bias into the review.

For example, clear inclusion criteria will avoid selecting studies according to whether

their results reflect a favorite conclusion.

Sometimes it is necessary to reformulate inclusion and exclusion criteria during the review process.

Creating a protocol is an important step in the review process.

It aims to reduce the impact of authors biases,

promotes transparency of methods and processes,

and reduces the potential for duplication.

The review protocol sets out the methods to be used in the review,

just like the protocol for a clinical trial.

Decisions about the review question, eligibility criteria,

search strategy, study selection, data extraction,

quality assessment, data synthesis,

and plans for dissemination should be addressed.

If modifications to the protocol are required,

these should be  clearly documented and justified.

Modifications may arise from a clearer understanding of the review question,

but should not be made because of an awareness

of the results of individual studies.

When you complete your protocol, you may want to consider registering it.

PROSPERO, from The Center for Reviews and Dissemination at the University of York,

is an international database of prospectively registered systematic reviews

in health and social sciences.

Key features from the review protocol are recorded and maintained as a permanent record.

PROSPERO aims to provide a comprehensive listing of systematic reviews,

registered at inception, to help avoid unplanned duplication

and enable comparison of reported review findings

with what was planned in the protocol.

Let's take a look at some existing protocols.

This is the PROSPERO database.

Let's go to the search bar and test a search.

There are five protocols in PROSPERO on diabetes management.

Let's take a look at one to get a better idea of what these protocols look like.

You can see at the beginning of general citation for the protocol,

and then it goes into some of the details it includes the review question being asked,

plans for the searches that they'll do,

information about the types of study that'll be included,

as well as everything that fits all of the PICO criteria,

the condition, interventions, comparisons, and outcomes.

If we scroll further down we see their plans for data extraction,

plans for data synthesis and analysis,

as well as how they plan to disseminate their systematic review

when it's completed.

It includes information about all of the team members,

as well as their start date, planned completion date,

and additional information.

It does also allow you to come in and include revisions.

So you can see the date of registration,

as well as the date of any updates

and they can also go in and 

indicate which steps they've completed in the process,

whether they've stared them,

or completed them or both.

As of March 2016, there were over 12,000 protocols registered in PROSPERO.

Once you have a protocol, you're ready to begin your search.

Searching the literature is an integral part of the systematic review.

It's also a much more involved process than you may be used to.

In order to be as comprehensive as possible searches are often broad in scope.

There's a balance to strike between finding all of the articles on a topic,

along with the number of irrelevant studies,

and finding only relevant articles,

but missing out on additional articles that would be of use.

In general you should expect a large number of results,

the exact amount will vary depending on the scope of your research question,

but you may see as few as a couple hundred results,

or as many as several thousand.

Be prepared to sift through a large number of citations.

The first step is to identify the databases you'll search.

The standard resources recommended by the Cochrane Handbook

are PubMed, Embase, and the Cochrane Central Register of Controlled Trials.

But depending on your topic, you may include others, as well.

Next, you'll identify search terms, both keywords and controlled vocabulary,

which will allow you to be as comprehensive as possible in your search.

After completing your search, you may want to include a search of the grey literature,

which is anything that isn't traditionally published,

such as conference proceedings, dissertations, and theses,

and government reports.

You can search resources like clinicaltrials.gov,

the ProQuest dissertations database,

or government agency websites for these.

You may also want to try hand searching specialized journals,

contacting experts in the field,

and searching the reference list of articles you already have.

It's very important to keep a record of all of your searches.

Not only should you report at least one database search strategy in your final report,

but you may want to update the review in the future

and you'll need to replicate your original search.

This is an example of a search and Embase from a review on treating insomnia

with cognitive behavior therapy.

You can see the careful attention to the terminology here,

as well as specific fields and tricks for the Embase database.

It's highly recommended that you work with a librarian

in constructing your systematic review searches.

In fact, The Institute of Medicine Standards strongly advocate

working with a librarian.

The HSHSL has a systematic review service available,

and we can work with you to put together all of your searches

for your systematic review.

Once you've collected all of your studies,

you'll need to screen them against your eligibility criteria.

This is a step to be completed by at least two members of the team.

A typical process for selecting studies for inclusion in a review is as follows:

First, remove duplicates.

This can be done in your citation management software,

and is generally completed by a librarian if you use our service.

Next, you'll go through each articles title and abstract,

and remove all obviously irrelevant citations.

Then you'll retrieve the full text of all of these articles,

and examine each one in relation to your inclusion and exclusion criteria.

In the end, the team can resolve any differences of opinion.

It's important to have a record of decisions made for each article.

Reproducibility guards against duplication of efforts,

makes it possible to justify your decisions,

and allows reviews to be updated more easily.

It may be helpful to develop an appropriate form

to help select and keep track of articles that meet eligibility criteria.

Reference management programs such as Refworks or Endnote

allow basic inclusion decisions to be made and recorded in custom fields.

Some people find spreadsheets to be just as easy at this stage.

If you need to record several comments or decisions,

a database program such as Microsoft Access may be useful.

There are also programs specifically designed for the study selection process,

which I'll talk about a bit more later.

The screening process is documented using the PRISMA flow chart,

which you may be familiar with from published systematic reviews.

This flow chart shows the number of studies remaining in each stage.

It's a simple and useful way of documenting the studies selection process.

The flow chart shows the initial search results from database searching,

as well as citations found from other sources.

Maybe a hand search of journals, or from going through reference lists.

These articles were screened, and 567 duplicates were removed.

After screening titles and abstracts  another 1470 articles were removed.

This is pretty typical.

In this case only 220 full-text articles  were assessed for eligibility.

This flowchart explicitly lists the reasons

for removing any articles excluded at this stage.

Finally, you'll see the number of studies included in the meta-analysis.

Sometimes you'll see separate numbers for articles included

in a qualitative analysis versus a quantitative analysis.

In this case, all remaining studies were included in both.

This flowchart should be included in your final report.

Once you've chosen your studies to include,

you can extract data from them.

Data extraction should be detailed in the protocol,

including what will be extracted and how.

More than one person should do this to reduce error and bias.

To standardize the process and improve the validity of results,

it's crucial to use the data extraction form.

Given the important functions of data collection forms,

ample time and thought should  be invested in their design.

Because each review is different, data collection forms will vary across reviews.

But, there are a lot of similarities in the types of information that are important,

and forms can be adapted from one review to the next.

First, consider how much information should be collected.

Collection of too little information, or omission of key data,

can lead to the need to return to study reports later in the review process,

which is really another waste of time.

Reviewers most commonly create their own data extraction forms

using Word, Excel, or Access,

depending on the complexity of the data and their own comfort with the software.

While you can create your own data extraction form to meet your needs,

you may want to consider using an existing tool to create a form.

EPPI-Reviewer and Distiller SR can be used during data extraction

Neither is free and there use may be cost-prohibitive,

but they are designed to assist with this stage of the systemic review process.

One free option is SRDR, the Systematic Review Data Repository,

created by researchers at the Brown University Evidence-based Practice Center

with support from AHRQ.

SRDR can facilitate data extraction and storage for systematic reviews or meta-analyses.

It's an open and searchable archive of systematic reviews and their data.

Next, you'll appraise the included studies.

Assessment of study quality gives an indication

of the strength of evidence provided by the review.

Ultimately, quality assessment helps answer the question

of whether the studies are robust enough  to guide treatment, prevention, diagnostic, or policy decisions.

In appraising the studies for your review,

you'll assess things like the risk of bias, the appropriateness of the study design,

the  quality of reporting, choice of outcome measures,

statistical issues, and generalizability.

Various tools are available to aid an assessment,

but there's no single tool that's suitable for use in all reviews.

Your choice should be guided by the study design and level of detail required.

Most appraisal tools have been developed for specific types of research study designs,

rather than specific types of research questions.

For example, there are tools for assessing randomized control trials

and others for qualitative studies.

When reviews include more than one type of study design,

separate lists can be used or combined list selected and developed.

Taking a broad view, the aim of assessing study quality

is to establish how near the truth its findings are likely to be,

and whether the findings are of relevance in the particular setting or patient group of interest.

As mentioned there are many tools for the critical appraisal of studies.

The grade system, grading of recommendations, assessment, development, and evaluation,

is a major one.

Over 60 organizations including the CDC,

World Health Organization,

the American College of Physicians, and the Cochrane Collaboration

have adopted the grade system.

You can find more information  about appraisal tools that are available,

as well as links to those tools, on our website.

The website links to more information about appraisal

including several different appraisal tools  that are available,


the next step. synthesis, involves the combination and summary of the findings

of individual studies included in the systematic review.

Synthesis can be done quantitatively using formal statistical techniques such as meta-analysis,

or a statistical pooling of results as inappropriate

through a narrative approach.

As well as drawing results together,

synthesis should consider the strength of evidence,

explore whether any observed effects  are consistent across studies,

and investigate possible reasons for any inconsistencies.

The planned approach for synthesis should be decided to the outset of the review,

depending on the type of question posed and the type of studies that are likely to be available.

Narrative and quantitative approaches are not mutually exclusive.

Components of narrative synthesis can be incorporated into a review

that's primarily quantitative in focus,

and those that take a primarily narrative approach can incorporate some statistical analyses,

such as calculating a common outcome statistic for each study.

A Project for the Economic and Social Research Council Methods Program

has developed general framework on the conduct of narrative synthesis and systematic reviews.

This involves presenting a theory of how the intervention works,

summarizing the findings of individual studies,

discussing the relationships between studies,

and finally providing an overall assessment of the  strength of the evidence.

All systematic reviews should contain text and tables

to provide an initial descriptive summary and explanation of the characteristics

and findings of the included studies.

We'll see an example of the table on the next slide.

In general, making recommendations  for practice doesn't fall within the purview of systematic reviews.

This is typically the domain of clinical practice guidelines.

Systematic review authors can make conclusions about the need for further research,

including the nature of further research that would be most effective.

Both quantitative and qualitative synthesis should begin

by constructing a clear descriptive summary of the included studies.

This is often presented in the form of a summary of findings table.

This is a summary of findings table from the same insomnia review we've seen previously.

The report should describe the review methods clearly

and in sufficient detail that others could repeat them if they wished.

Commissioning bodies and journals usually have specific requirements

regarding presentation and layout that should be followed when preparing a report or article.

Many of them will ask you to follow the PRISMA guidelines.

The PRISMA checklist provides guidance on the information to include in your report.

The EQUATOR Network is an initiative that seeks to improve the quality

of scientific publications by promoting transparent and accurate reporting of health research.

It provides reporting guidelines for various types of studies,

and information for authors of research reports.

Many tools have been developed to assist systematic reviewers.

Not all of these tools will be free.

Distiller SR is software design for the screening and data extraction phases of the systematic review.

EPPI-Reviewer is a web application that allows for management of all stages of the review.

Covidence is a tool for screening and data extraction.

It's used in partnership with the Cochrane Collaboration to support Cochrane reviews.

RevMan is software from the Cochrane Collaboration as well,

it's used for preparing and maintaining Cochrane reviews.

It's required for those conducting Cochrane reviews,

but you can use it free of charge for purely academic purposes, as well see.

CMA is a software that will perform your meta-analysis quickly,

computes treatment effects, and creates forest plots.

MetaLight is a software application designed to support the teaching and learning of meta-analysis.

It's available freely from EPI Center.

You can find links to these tools and others on the HSHSL systematic review guide.

Before we conclude, here's some additional reading that may be useful

as you begin working on a systematic review.

I want to wrap up by directing you to the library's systematic review service.

Librarians at the HSHSL are experts searchers who can support faculty investigators

in conducting comprehensive literature searches for systematic reviews.

Librarians can determine if a systematic review has already been done on your topic,

recommend specific databases and gray  literature sources to be searched,

collaborate with investigators to develop appropriate search strategies

to ensure that all relevant studies are identified,

conduct the literature search in relevant databases,

deliver search results into a citation management tool like RefWorks,

assist in writing the search methodology section of the review,

and setup search alerts and follow up with updates as needed.

You can also find more information about tools researchers use

at various points in the process on our website.

The HSHSL systematic review website will walk you through more details

about the library systematic review service.

You can also find more information about systematic reviews here.

This includes links to various standards  mentioned earlier in this presentation,

such as PRISMA, The Cochrane Handbook,

The Institute of Medicine guidelines, and more.

You'll also find links here to places to register your protocol,

including PROSPERO.

This is also where we saw earlier a list of critical appraisal tools.

The tools page provides links to many of  the software tools mentioned earlier in this presentation.

You can also find a page about other types of reviews

should you decide that a systematic review is not the best approach for your particular topic.

Finally, you'll find a link to our request form.

This is where you can request a consultation with the librarian at the HSHSL

to learn more about our service, and to  begin the search process.

Thanks for listening to the introduction of conducting systematic reviews

from the Health Sciences and Human Services Library.

If you have any questions please go to the website on this slide

for more information.

You can fill out a request  form if you'd like us to assist with your systematic review,

or you can get in touch with the librarian for your school for additional questions.