Research Methods & Reporting

Enhancing the usability of systematic reviews by improving the consideration and description of interventions

BMJ 2017; 358 doi: https://doi.org/10.1136/bmj.j2998 (Published 20 July 2017) Cite this as: BMJ 2017;358:j2998
  1. Tammy C Hoffmann, professor of clinical epidemiology1,
  2. Andrew D Oxman, research director2,
  3. John PA Ioannidis, professor3,
  4. David Moher, senior scientist4,
  5. Toby J Lasserson, senior editor5,
  6. David I Tovey, editor in chief5,
  7. Ken Stein, professor of public health6,
  8. Katy Sutcliffe, senior research associate7,
  9. Philippe Ravaud, professor of epidemiology8,
  10. Douglas G Altman, professor of statistics in medicine9,
  11. Rafael Perera, professor of medical statistics10,
  12. Paul Glasziou, professor of evidence based medicine1
  1. 1Centre for Research in Evidence-Based Practice, Faculty of Health Sciences and Medicine, Bond University, Queensland, Australia
  2. 2Centre for Informed Health Choice, Norwegian Institute of Public Health, Oslo, Norway
  3. 3Departments of Medicine, of Health Research and Policy, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA
  4. 4Ottawa Hospital Research Institute, Ottawa, Canada,
  5. 5Cochrane Editorial Unit, Cochrane Central Executive, London, UK
  6. 6Institute for Health Research, University of Exeter Medical School, Exeter, Devon, UK
  7. 7EPPI-Centre (Evidence for Policy and Practice Information and Co-ordinating Centre), Institute of Education, University College London
  8. 8Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France; and Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France
  9. 9Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK
  10. 10Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  1. Correspondence to: T Hoffmann thoffmann{at}bond.edu.au
  2. Accepted: 5 June 2017

The importance of adequate intervention descriptions in minimising research waste and improving research usability and reproducibility has gained attention in the past few years. Nearly all focus to date has been on intervention reporting in randomised trials. Yet clinicians are encouraged to use systematic reviews, whenever available, rather than single trials to inform their practice. This article explores the problem and implications of incomplete intervention details during the planning, conduct, and reporting of systematic reviews and makes recommendations for review authors, peer reviewers, and journal editors

Summary points

  • Intervention details are rarely fully considered or completely reported in systematic reviews, limiting the reproducibility and usability of systematic reviews—this is wasteful

  • Intervention details are needed in many stages of the review process—from question formulation to decisions about eligibility and analyses, results interpretation, and use of the review findings

  • Systematic review authors should give careful consideration to intervention details during the planning, conduct, and reporting of the review, including extracting, requesting, and fully reporting them

  • Improving the consideration and description of interventions in systematic reviews, such by providing a summary table with details, is likely to contribute to reducing avoidable waste in health research

Up to 60% of interventions in trial reports are inadequately described, although more information can sometimes be obtained from the authors.1 When interventions are inadequately described in randomised trials, clinicians and patients must guess how to use effective interventions, and researchers are unable to replicate or build on the research. Another consequence is that the intervention details are not available to the authors of systematic reviews.

Few studies have examined the problem of inadequate description of interventions in systematic reviews. An analysis of 58 systematic reviews of stroke interventions2 found that most were missing information for the majority of items needed to make an intervention description adequate. For example, details such as the intervention procedure, materials, fidelity, and tailoring were missing from more than 80% of reviews.

Inadequate intervention reporting in trials not only produces avoidable waste for the original trials but is compounded in downstream uses of the trials, such as in systematic reviews, with implications for the reproducibility and usability of the systematic review.

Appropriate use of intervention details in the planning, conduct, and reporting of systematic reviews is facilitated by interventions being well described in trials and other evaluative studies. The Template of Intervention Description and Replication (TIDieR) checklist and guide was developed and published in 2014 to help authors comprehensively describe interventions, with an initial focus on trials.3 Historically, the development of systematic review techniques, methods, and technologies has focused on aspects such as searching, assessing and reporting risk of bias, and statistical methods. The clinical usability of the results of systematic reviews has had less attention, and the reporting of intervention in reviews almost none.4

To identify a common approach for improving the consideration and reporting of intervention details in systematic reviews a group of experts—including systematic review authors, trial authors, journal editors, methodologists, and statisticians with expertise in intervention descriptions, reporting guidelines, trials, and systematic reviews—attended a one day meeting in Oxford in June 2016. Representatives from the following groups also attended: the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) group,5 the Cochrane Library, the EQUATOR Network, the Template of Intervention Description and Replication (TIDieR) group,3 the Evidence for Policy and Practice Information and Coordinating (EPPI) Centre, and the NIHR Journals Library. The meeting organisers (TH, PG) invited participants, drafted the agenda, invited presentations, and collected and disseminated background literature. The day consisted of stimulus presentations on key relevant topics and associated research followed by group discussions and recording of discussion points and possible recommendations. In the final session, delegates discussed and collaboratively modified draft recommendations to improve the consideration and description of interventions in systematic reviews until group consensus was achieved. After the meeting, the group members (authors of this paper) refined the final wording of these recommendations, which are reported here.

Recommendations to improve the consideration of interventions when planning, conducting, and reporting systematic reviews

The recommendations that authors of systematic reviews should undertake when planning, conducting, and reporting their reviews are shown in box 1. An elaboration and explanation of each recommendation follows the box. They are applicable to all systematic reviews of studies of intervention effectiveness. Specific recommendations for Cochrane reviews and non-Cochrane reviews are detailed later in this section. For most systematic reviews, many of the recommendations also apply to the comparator intervention with these details needing appropriate consideration and reporting as well.

Box 1: Recommendations for authors to improve the consideration interventions when planning, conducting, and reporting systematic reviews

Planning the review

1 Consider intervention details during question formulation

Use TIDieR3 to identify any important details of the intervention that will determine the questions that the review will ask, including how broad or narrow the review should be and what the main comparison will be.

2 Describe intervention considerations in the review protocol

Describe the intervention and its relevant components (if multicomponent) and characteristics in the protocol. Relevant protocol sections might include: the review question, background, search terms, eligibility criteria, data items, and quantitative synthesis plans.

Conducting the review

3 Extract intervention details as part of data extraction

Use TIDieR as a guide to the essential intervention characteristics to include in the data extraction form and extract accordingly.

4 Request missing intervention details

When feasible, request missing details from the authors, using TIDieR as a guide to which details to request, and note when details are not available.

5 Consider intervention characteristics during statistical analyses and exploration of heterogeneity when appropriate

Where appropriate and feasible, consider intervention characteristics as specified in the protocol when grouping studies, conducting analyses, and exploring heterogeneity.

Reporting the review

6 Report intervention details in a summary table

Provide a table that summarises the intervention details for each study (see template in web extra 1 and example in table 1).

7 Share intervention materials where possible

Where intervention materials are available, share or provide their location details in the review’s intervention summary table.

8 Describe implications for future research

If the summary of intervention details reveals important gaps in existing research or if the analyses identify a significant association between effect and the presence or absence of intervention components or characteristics, then describe the future research implications of this in the review.

Recommendation 1—Consider intervention details during question formulation

Many systematic review authors use the PICO format (patient, population or problem; intervention; comparison intervention (if appropriate); outcome of focus) to design their review question. When doing this, the I (intervention, and where necessary, its characteristics; and if a multicomponent intervention, the major components) should be given as much consideration as the other parts. Authors should use TIDieR to identify any important details of the intervention that should determine the questions that the review will aim to answer; for example, which active components are used, the timing of the intervention, the dose, the mode of delivery, or who provides the intervention. Such details will also help to inform the breadth of the review. If a scoping exercise was performed as part of the planning process, summarising the intervention details (such as in a summary table) from studies found in the scoping exercise might help inform this decision. Authors should also carefully consider intervention details when deciding on the main comparison that will be made in the review.

Recommendation 2—Describe intervention considerations in the review protocol

When registering a systematic review title (such as at PROSPERO; www.crd.york.ac.uk/PROSPERO/) and writing a protocol, authors should carefully consider and describe the intervention and its relevant components (if multicomponent) and characteristics. Items in the reporting guideline for systematic review protocols (PRISMA-P) that are particularly relevant to this include: item 7, explicit statement of the review question; 8, eligibility criteria; 10, search strategy; 12, data items; and 15a, criteria for quantitative synthesis.6 Further details about sections of the protocol relevant to intervention details are provided below:

Background

If relevant, protocol authors should report how consideration of details of the intervention affected the scope of the review and the categorisation of interventions within this scope. Where relevant, authors should also clarify why differences in the details of the intervention might modify its effects; for example, which active components are used, the timing of the intervention, the dose, the mode of delivery, or who provides the intervention.

Objectives

Intervention details might determine the main comparisons that will be made and should be considered when deciding on the review’s objectives.

Eligibility criteria

Intervention details might be part of inclusion or exclusion criteria and should be clearly stated. When intervention details in potentially eligible studies are not stated or not clear, this step in a review can be compromised.

Data extraction

Protocols should include plans for collecting sufficient details about the interventions so that they can be described adequately. TIDieR items can be used as a guide to which intervention characteristics should be incorporated into the data extraction form.

Missing information

Because trial reports often do not adequately describe interventions but trial authors can provide missing details,17 review authors should plan at the protocol stage to request missing intervention details from investigators.

Statistical analyses, such as subgroup, dose-response, and meta-regression

Decisions about appropriate inclusion and grouping of studies for analyses often requires knowledge of the characteristics of the interventions that were studied. When there is a reason to believe that differences in intervention characteristics (for example, the dose) might lead to different effects, these differences should be identified in the protocol, together with the basis for the assumptions that they might modify the effect, the expected direction of effect modification, and a plan for undertaking a subgroup analysis or sensitivity analysis. In network meta-analyses, creating nodes can be difficult if the interventions are not sufficiently described.

Recommendation 3—Extract intervention details as part of the data extraction process

As specified in the protocol, in the data extraction stage, review authors should extract details of the essential intervention characteristics (guided by TIDieR items) for each included study.

Recommendation 4—Request missing intervention details

If, after extracting intervention details from the primary studies and other available sources (such as online supplements or trial websites), intervention details are missing, review authors should request the missing details from the authors, where feasible. When review authors attempt to contact trial authors and either do not receive a response or find that intervention details are unable to be shared, this should be noted in the review to alert readers This might inform their choice of intervention and also save them from trying to obtain details in vain.

Recommendation 5—Consider intervention characteristics during statistical analyses and exploration of heterogeneity when appropriate

When considering reasons for heterogeneity in review results, having sufficient information about the characteristics of the interventions evaluated might be very important. Where appropriate, decisions about grouping studies and conducting analyses should incorporate knowledge of intervention details as specified in the protocol.

Recommendation 6—Report intervention details in a summary table

Review authors should provide a table that summarises the intervention details for each study (see example in table 1 and the blank table provided as a template in web extra 1). The column headings are based on the TIDieR items. A summary table serves a few purposes; it helps readers to compare the characteristics of the interventions and consider those that may be feasible for implementation in their setting; it highlights interventions that have missing or unavailable details; it shows which trials did not specify certain characteristics as part of the intervention; and it highlights characteristics that have not been studied in existing trials.

Table 1

Example of table summarising intervention details (for each TIDieR item3) in a systematic review (from Coxeter et al8)

View this table:

Review authors should list all trials and not omit those that provided evidence that a certain intervention was not effective. Knowing the details of an intervention that was not effective can inform future research. Moreover, it is helpful for readers to know that a particular implementation of the intervention in a specific context or when compared to a specific control did not work—context might be particularly important for non-drug interventions.

Recommendation 7—Share intervention materials

During the review process, the authors might gather intervention materials, such as educational materials provided to trial participants as part of the intervention from trial authors. These materials are the most common missing element of intervention descriptions,1 even though interventions cannot be faithfully implemented without them. If review authors have obtained permission to do so, these materials should be deposited in online repositories (such as Figshare, Dryad, Open Science Framework, or OpenTrials) or uploaded as online supplementary materials of the review, and their availability and location should be indicated in the intervention details table in the review.

Recommendation 8—Describe implications for future research

Review authors should summarise the intervention details of included studies (such as in table 1). If this summary reveals important gaps in existing research; for example, if no or few interventions used a particular component (for multicomponent interventions) or dose (or intensity for non-drug interventions) or delivery method, this should inform the future research section of the review. Similarly, if analyses conducted in the review show that particular characteristics or components of the intervention were (or were not) significantly associated with effect, this can also inform future research. Most of the time, the heterogeneity in effect sizes that might be explained by one or more specific characteristics of an intervention is not definitive, as such assessments are generally confounded by other study features. In the discussion section of the review, authors should consider and justify the extent to which the review findings support conclusions about whether any of the differences in intervention details lead to important differences in effects.910

Cochrane reviews

Authors of Cochrane intervention reviews should follow the Methodological Expectations for Cochrane Intervention Reviews (MECIR) standards. The revised MECIR standards published in October 201611 reference TIDieR as a guide for collecting and reporting intervention characteristics (Standards C44 and R65). Information about TIDieR has also been added to the training materials for Cochrane authors.12 Cochrane authors are encouraged to provide a structured account of intervention details in the table of “Characteristics of included studies.” They can provide an additional summary table with intervention details for each study (as shown in table 1, which comes from a Cochrane review8) and can share intervention materials gathered during the review (see recommendation 7 and box 1) as appendices to the review.

Non-Cochrane reviews

Authors of non-Cochrane reviews are encouraged to follow the recommendations listed in box 1. The relevant PRISMA-P items are listed earlier in recommendation 2. The relevant PRISMA items include: 1, title; 2, abstract; 3, rationale; 4, objectives; 6, eligibility criteria; 8, search; 9, study selection; 10, data collection process; 11, data items; 18, study characteristics; 25, limitations; and 26, conclusion and future research. Modification of guidance for the relevant PRISMA5 and PRISMA-P6 items will be considered when these reporting guidelines are next updated.

Recommendations for peer reviewers and editors of systematic reviews

As with other research replicability and reporting issues, peer reviewers and editors also have a role to play in helping to ensure that interventions are appropriately considered and reported in systematic reviews. They should be guided by many of the recommendations in box 1 and should check that interventions are clearly defined and their details are appropriately considered in analyses, are reported as completely as possible, and are considered in the review’s discussion, conclusions, and, where appropriate, the future research section.

Using the findings of a systematic review: the importance of knowing intervention details

New trials should be designed according to what is already known from systematic reviews.13 Providing complete intervention descriptions in systematic reviews is important for informing researchers as they develop and modify interventions to evaluate in future studies (see recommendation 8).

Clinicians, patients, and policy makers cannot implement effective interventions if details of the interventions are not known. Review users should be able to compare the details of the interventions and consider whether—and, if so, how—to implement interventions in their setting (see details in the elaboration of recommendation 6 and section below). As well as individual decisions, having appropriate intervention details might also influence broader decisions, such as those about reimbursement or adapting standard practices. The usability of many downstream evidence resources that incorporate systematic review findings (such as clinical guidelines and patient decision aids) is also influenced by whether the interventions are appropriately detailed in the review. The safety of an intervention can also be compromised if there is not transparency about all its characteristics.

Choosing which intervention to implement

We do not intend to provide guidance about methods for selecting interventions for clinical implementation from those included in a systematic review. Such decisions need to be informed by multiple considerations14 including the size of the desirable effects; the size of the undesirable effects; the balance between the desirable and undesirable effects (considering patients’ preferences and how much people value the main outcomes); the certainty of the evidence; resource requirements; cost effectiveness; impacts on equity; intervention feasibility and acceptability; and the availability of intervention details. Because these considerations go beyond the evidence that is included in most systematic reviews, and as there is no optimal method of selecting a particular intervention from those included in a review, in most circumstances it is not appropriate for review authors to nominate a single recommended intervention. Details of approaches for choosing an intervention are described elsewhere.1415 But all approaches require detailed descriptions of the intervention, and some also require detailed descriptions of the comparator interventions.

Although review authors generally should not make recommendations about a single intervention, they might want to provide a summary paragraph of the known factors to consider when choosing an intervention. This may be particularly helpful if users of the review choose to follow a “single trial based choice” approach.15 In this approach, users examine the trials and consider the effects (benefits and harms) and risk of bias of single studies; then they consider the context, feasibility, and requirements of the various interventions. A summary table of intervention details (such as in the example in table 1) might help the user with this step. While the information that needs to be considered and summarised will depend on the intervention being reviewed, an example of the broad content that a summary paragraph in a review might follow is: “Among the [number of] trials, there are [number of] trials that have a low risk of bias and have sufficiently described interventions. All of these involved [list common characteristics], but there are a number of variations to consider, depending on [cost, time, risk of harms, training requirements, availability, and so on].”

Further research

Many aspects of using and reporting intervention details in systematic reviews need further research. For example, studies should explore methods for reporting intervention details and for incorporating intervention details into forest plots so that effect sizes, risk of bias, intervention characteristics, and availability of intervention details can be considered simultaneously. Incorporating intervention details into the conduct and presentation of overviews and network meta-analyses16 also needs exploring. The extent to which review authors make changes to the scope of eligible interventions (and how broad or narrow this is) as reviews progress from registration, to protocol, to a published review is not known. More complete intervention reporting at each of these stages of a systematic review is necessary to progress this research agenda. Research with end users of reviews (including clinicians, patients, guideline developers, and policy makers) to better understand how they use review results and which details influence their choice when deciding between interventions would also be valuable. Further research is also needed into approaches, such as Qualitative Comparative Analysis17 and logic models,18 for identifying which configurations of intervention characteristics and contextual features19 are critical for successful outcomes.

Conclusion

Improving the completeness of intervention descriptions in systematic reviews is likely to be a cost effective contribution towards facilitating evidence implementation from reviews and reducing the research waste that is caused by reviews failing to consider and provide sufficient details about interventions. With implications for being able to reproduce and implement systematic reviews, everyone with a role in producing, reviewing, and publishing systematic reviews should commit to helping to solve this remediable barrier.

Footnotes

  • Contributors: TCH initiated a meeting of all authors in Oxford in June 2016 and led the writing of the paper. All authors participated in discussions at the meeting and contributed to the drafting and revision of the paper and approved the final version. Each of the authors has expertise in intervention descriptions, reporting guidelines, and/or conducting trials and systematic reviews. TCH is the guarantor.

  • Funding: There was no funding for the development of this paper. PG is supported by a National Health and Medical Research Council of Australia Research Fellowship. JI is supported by the Meta-Research Innovation Centre at Stanford (METRICS), which is funded by a grant from the Laura and John Arnold Foundation. DM is supported by a University Research Chair, University of Ottawa.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and declare the following interests: TH, PG, DM, DA, and RP are members of the team that developed the TIDieR guide. DM led development of PRISMA and PRISMA-P. DA, DM, PR, and PG are directors of the EQUATOR Centres in Oxford, Ottawa, France, and Australia, respectively.

References

View Abstract