Analysis

Allocation concealment in randomised controlled trials: are we getting better?

BMJ 2016; 355 doi: https://doi.org/10.1136/bmj.i5663 (Published 17 November 2016) Cite this as: BMJ 2016;355:i5663
  1. Laura Clark, research fellow,
  2. Caroline Fairhurst, research fellow,
  3. David J Torgerson, director
  1. York Trials Unit, University of York, York YO10 5DD, UK
  1. Correspondence to: laura.clark{at}york.ac.uk
  2. Accepted 18 October 2016

Laura Clark and colleagues assess the allocation concealment methods in a sample of randomised controlled trial publications

A robust randomised controlled trial (RCT) must use allocation concealment—that is, separate the act of randomisation from the person recruiting participants. Poor randomisation methods cause exaggerated treatment effects, are open to subversion by researchers or clinicians, and have a knock-on effect on systematic reviews.1 2 3

The CONSORT statement, which leading medical journals endorse, states that the method of allocation (comprising sequence generation, allocation concealment mechanism, and implementation) should be clearly described.4 Allocation concealment is dependent on the method of sequence generation as well as the concealment mechanism.

Almost a fifth of trials published in major medical journals in 2002 used inadequate concealment, and a quarter failed to describe how the allocation was concealed.2 Here we examine a sample of RCTs published in 2015 to see whether the situation has improved.

Defining inadequate allocation concealment

We searched four high impact medical journals (The BMJ, Journal of the American Medical Association (JAMA); the Lancet, and the New England Journal of Medicine (NEJM)) and found 79 RCTs published between June and August 2015. We extracted and judged their mechanism for allocation concealment, taking into consideration the study design, sequence generation method, and allocation concealment mechanism. We defined an inadequate process as one that used envelopes as the method of allocation concealment (box 1) or used stratified block randomisation by site with small block sizes as the sequence generation method (box 2), except in double blind trials. If insufficient detail was provided in the paper, we checked the protocol or emailed the authors.

Box 1: Sequentially numbered opaque sealed envelopes (SNOSE)

Envelopes containing the treatment allocation are opened by the recruiting clinician on participant enrolment. To be robust, the envelopes should be truly opaque, sequentially numbered, and opened in the correct order. The clinician should not open the envelope in advance and should ensure that the envelope seal has not been broken. Even in these circumstances we cannot guarantee that envelopes have not been opened in advance to allow strategic scheduling of patient appointments to match the recruiter’s preferred allocation. A surgical trial found that three of five surgeons had opened envelopes in order to subvert the randomisation.5 Trials that use SNOSE are more likely to show a statistically significant treatment effect than trials that use more secure allocation methods, such as web based or telephone randomisation.2 Despite this, the Cochrane handbook of systematic reviews says that trials that use SNOSE have a low risk of bias.6 In practice, the bias risk is only lessened or eliminated when the people with access to the envelopes are distinct from those recruiting participants to the trial.

Box 2: Block randomisation

Most trials use restricted randomisation methods to generate the allocation sequence, such as stratification,7 which requires the use of block allocation within each strata. In this method a limited sequence of allocations are repeated: for example, a block size of four with two treatments (A and B) has six potential blocks of sequences (AABB, ABAB, BBAA, BABA, ABBA, and BAAB). Even if a robust allocation concealment mechanism is used (such as central web based randomisation), subversion of the allocation is possible. For example, stratifying by site and using a small fixed block size makes the allocation sequence predictable7—in a two arm trial using randomisation stratified by site and a fixed block size of four, every fourth allocation can be accurately predicted, and the third allocation in a block can be predicted a third of the time, if one keeps a record of the previous allocations given to patients. Only simple randomisation avoids the problem.8 This is not just a theoretical concern. In a trial of supplemental oxygen for retinopathy of prematurity, clinicians tried to avoid recruiting and allocating patients to the control group, which they judged as being undesirable for some patients.9 An RCT of rehabilitation for patients with fractured neck of femur failed because, despite using telephone randomisation, the block size of six was deciphered part way through the trial, which led to prediction and selection bias.10 The use of larger or variable block sizes or avoiding stratifying by site can minimise this problem. Stratification is only beneficial for small sample sizes or slowly recruiting trials or if treatment logistics demand some predictability of treatment volume (for example, surgical treatments). For many, if not most, trials simple randomisation is preferred (if n>100).7

Fifteen trials were poorly randomised

Twenty seven (34%) of the RCTs were placebo controlled double blind trials, in which allocation is generally well concealed; participants are assigned a number corresponding to a packet of drugs, and only the pharmacist has access to the unblinding codes. One of these trials used envelopes, but as the pharmacist opened the envelopes after the clinician enrolled the participant it was deemed adequate. We judged these trials, and 22 (28%) of the remaining trials, as adequate. We initially found that 13 trials (16%) had used a randomisation method that put them at risk of bias and 17 trials (22%) contained insufficient detail to determine whether the method of allocation concealment was adequate. We received more information from the authors of nine (53%) of these trials, two of which were found to be inadequate, giving a total of 15 (19%) trials with inadequate concealment (table 1). Seven trials used envelopes to allocate participants, seven trials used small block sizes and/or stratified by site, and one used both small blocks and envelopes.

Table 1

Trials with inadequate concealment

View this table:

We noted two inconsistencies with the use of block randomisation. The trial by Senn et al had an imbalance of 12 participants between the randomised groups; the largest possible imbalance for a block size of two stratified by three centres is three. Correspondence with the authors confirmed that 10 cases were misallocated to the intervention group.26 The authors said that a combination of technical and human errors accounted for the imbalance—“the laptop froze during randomisation, the server went down temporarily, research assistants inadvertently practiced on a live site, and participants went to the wrong session.” They reassigned the 10 participants to their originally assigned groups and found no change in benefit for the intervention.

The trial by Cox and colleagues27 said that patients were allocated 2:1 with blocks of four, but with a 2:1 ratio the block size should be divisible by three. Correspondence with the author confirmed that the statement in the paper was a mistake, and an erratum has been published in the Lancet.

More rigour is needed

Our findings—that 19% of trials described inadequate methods of allocation concealment and 22% failed to report their randomisation methods clearly—are similar to those found in 2002 (18% and 26% were inadequate and unclear, respectively). The sample size of the 2002 study was much larger (n=234) than here (n=79).

Despite the known inadequacies of the use of envelopes for treatment allocation, we found their use in at least nine trials. Regardless of whether the allocation code is computer generated or the investigators are blinded, the person enrolling the patient can potentially open an envelope in advance. In certain scenarios, such as trials conducted in remote areas, the use of SNOSE may be the only feasible approach—if so, envelopes should be sequentially numbered and it should be stated clearly that a person who is separate from the assessment and recruitment of the patient opens the envelope, as described in the CONSORT statement (item 10).4 Rigour might be further enhanced by writing patients’ details on the outside of envelopes that contain carbon paper, so that the patients’ names are transferred to the sheet of paper with the allocation on before the envelope is opened.

Researchers and funders must try harder (box 3). Minimisation is an alternative to block randomisation, which allocates participants to the trial arm that best maintains balance across specified stratifying factors.28 Minimisation is a dynamic mode of random allocation—instead of using an allocation list that is generated before the trial begins, a participant’s allocation depends on their characteristics and, crucially, the characteristics of the participants already enrolled. Allocation by minimisation is much more difficult to predict than stratified, block randomisation, particularly if a random element is introduced (that is, instead of using minimisation to completely determine the allocations, a pre-specified probability that the treatment will be chosen is used for each randomisation).

Box 3: Advice for researchers and funders

Advice for researchers
  • Consult with a methodologist when designing an RCT

  • Use simple randomisation for larger sample sizes and consider using minimisation if stratification is required

  • When calculating costings for an RCT, factor in funding for third party randomisation services

  • If SNOSE are the only practical method to conceal the randomisation sequence ensure that their preparation and execution is methodologically rigorous—state who prepared the envelopes, what (if any) additional security measures were in place, and who opened the envelopes in the publication of the RCT

  • Explicitly state randomisation and allocation concealment methods in protocol and publication. If there is a reason why the information cannot be provided in the publication (for example, the word limit) refer to the section of the protocol in which a reader can find it, or supply the information in a supplementary document

Advice for funders
  • Ensure a methodologist is involved in designing and reporting the RCT

  • Do not fund RCTs using SNOSE unless justified as the only available option

  • Do not fund research that uses a block size of less than six with site stratification

Leading medical journals should specify that, at a specified point in the future, trials using small block sizes or SNOSE will not be accepted for publication, except for in extenuating circumstances, such as in emergency medicine or based on the location of the trial (for example, in remote areas that don’t have access to the internet or a telephone system). If these methods are used the details should be described explicitly and transparently. Meanwhile, journals could request that completed trials that used blocked randomisation perform a Berger-Exner test29 to assess for selection bias. This statistical test looks for a relationship between baseline variables and the position of the participant in the block. In the absence of subversion no relation should be found between these factors. This is not onerous for the authors and would demonstrate that their trial was conducted with robust methodology. Refusal to perform the test would preclude publication. Some trials30 31 that use block randomisation do report checking for subversion, but this statistical approach is rarely used.

Journals should ask a methodologist to carefully review the reported methods of randomisation before any trial is published or even sent for review. If those planning future trials felt that publication in a high impact journal was closed to them if their randomisation system was not robust they might be more focused at the design stage. It would also lead to an awareness of, and education in, the importance of robust randomisation methods and clearer reporting. Although journals endorse CONSORT, they are not enforcing it properly. We propose that, for randomisation at least, an editor or reviewer is mandated to ensure that the text is compliant with CONSORT.

Key messages

  • Good allocation concealment is vital for robust randomised trials

  • 38% of the trials in our sample of those published in major medical journals did not report good allocation concealment methods

  • Journals should insist that sealed envelopes and other weak concealment methods are no longer used

Footnotes

  • Contributors and sources: DT conceived of the original idea, LC identified trials, and CF, DT, and LC extracted data from the trials. DT jointly wrote the first draft with LC, and CF contributed to revising the draft. LC is the guarantor.

  • Competing interests: We have read and understood the BMJ policy on declaration of interests and declare that we have no competing interests.

  • We thank Doug Altman for his helpful comments.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract

Sign in

Log in through your institution

Free trial

Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
Sign up for a free trial

Subscribe