To The Editor, BMJ
Dear Dr Godlee
Balen et al have responded to our papers. [1,2] In our BMJ open paper we systematically identified 276 claims of benefit relating to 41 different fertility interventions and 16 published references were cited 21 times on 13 of the 74 websites we searched. In our BMJ analysis paper, we systematically examined the evidence for ‘38 additional fertility interventions’ and found evidence of improvements in live birth rates for only five interventions. We used standard critical appraisal techniques (as detailed in our paper) to explore the quality of these studies. We identified that for all five of these interventions the studies had methodological problems, raising uncertainty about the significance of the results. This is a serious issue for patients, public health, public trust, and regulators.
Balen et al suggest that our evidence review included - and found evidence lacking for - things which are not “add-ons”. We classified ‘add-ons’ in the BMJ analysis article in response to peer review where the issue was discussed (see online peer review comments).
Peer review comment: ‘The methodology has led them to five interventions for which there may be some evidence of improvement in live birth rates. One of these is intrauterine insemination in a natural cycle. This is an alternative treatment, rather than an “add-on”, and usually not one applied to the population who require IVF, and certainly not one intended to increase live birth rate.’
Our response: ‘We have also added a further table 1: Classifications of IVF Interventions: This is the only reference in the HFEA we could find on the words ‘add-on’ which was only discussed at the Feb 2016 meeting: 'The committee discussed which techniques are more common or established and may be incorporated into other sections of the website. The committee agreed that the remaining techniques for discussion could be placed onto a page about ‘Treatment add-ons’. And we note there is no current definition of what is an ‘Add-on’ treatment as it very much depends on what [the] clinic offers.'
We, therefore, classified 27 of these as ‘Add-on’ interventions: six as alternative treatments to IVF and five treatments for the preservation of fertility (see Table 1 in the paper). We, therefore, followed the advice of peer review. By ‘we’ we mean two general practitioners adjudicated and all authors confirmed the classifcation. Of note, there were concerns that we are not fit to do this. However, the HFEA’s advice is that when it comes to IVF “you should talk to your GP to go through the options available”.
Also in response to peer review, we added to our conclusion the following: ‘Clinics vary significantly in what they offer and there is a lack of clarity on what actually constitutes an ‘Add-on’ intervention. Clear classification systems could aid understanding and reduce variation in what is offered to couples.’
I also draw your attention to the HFEA Press statement at the time that did not question our classification. HFEA statement on fertility treatment 'add-ons' 28 November 2016 (http://www.hfea.gov.uk/10541.html)
HFEA official response to BBC Panorama regarding fertility treatment 'add-ons'.
‘UK fertility clinics generally provide excellent patient care. However, an increasing number of patients tell us they feel unsure about IVF treatment add-ons – whether they work and are worth the extra cost. This puts pressure on patients to make difficult decisions at what is already a stressful time for them.
'We are concerned about the recent step change in the use of treatment add-ons. Unfortunately, we have limited powers to stop clinics offering them, nor to control pricing. Instead, we publish information directly for patients, so that they can have the facts at their fingertips before they go to a clinic and are able to make informed decisions and properly consent to treatment.
'With expert advice from our scientific committee, we have for some years been publishing information about add-ons such as reproductive immunology and PGS, making it clear whether or not they are proven to be effective. We are also working on producing information for our new website about a wider range of add-ons, introducing a simple labeling system to highlight their levels of effectiveness and potential harm.’
Balen et al, therefore, appear to disagree with our categorisation of the 27 interventions as add-ons (see Box 2 of the paper).  This is a matter for reasonable debate, but more importantly suggests a lack of outward clarity to many users of this information, notably patients. We stand by our methods (we searched systematically for a classification system at the outset), and to the best of our knowledge, The BMJ is the only journal to have a published classification system.
Regardless, of whether Balen et al prefer to categorise interventions as “add-ons” or part of core treatment, this does not affect our conclusion: there is little or no evidence supporting interventions available from UK fertility clinics to improve live birth rate. We note that this is not a conclusion that is directly challenged by Balen et al.
Balen et al criticise our research team on the grounds that we are evidence based medicine researchers, clinicians, and epidemiologists, not “specialists in reproductive medicine”. This would seemingly seem an unusual line of argument. Given the known concerns over potential conflicts of interests in the IVF industry, and the extensive list of conflicts highlighted at the end of Balen et al’s letter, it was paramount that this work was conducted by an independent group with no ties to the IVF industry.  That said, should the opportunity arise for further work assessing the quality of evidence for fertility interventions we would be happy to consider collaborating with co-signatories to improve the evidence-base.
Balen et al appear to object to our use of “live birth” as the core metric of success. Once again, this appears to be an unusual line of argument to take. We refer Balen et al to our recent paper in Trials Journal entitled “Why clinical trial outcomes fail to translate into benefits for patients”.  In this paper we highlight some of the issues with substantial amounts of clinical research utilising surrogate, composite and subjective endpoints and failing to take account of patients’ perspectives when designing research outcomes. We agree with Barnhart, who highlights the need for the reproductive industry to make live births as the correct, patient focused, outcome for trials evaluating therapies for infertility.  We were therefore pleased to see that live birth rates are being used as a key outcome by RCTs currently recruiting patients in the UK.
Balen et al say that the sector has been well regulated by the HFEA. We question the strength of their argument to support this. Following the publication of our paper, we were pleased to note that the HFEA themselves have committed to updating their evidence to ‘provide free, clear and impartial information to all affected by assisted reproduction,’ and to provide clear evidence-based information to patients and the public as stated in their 2017 innovation and regulation plan.  We also noted that the Director of Strategy at the HFEA, Juliet Tizzard, publicly stated, in reference to our work, that the HFEA “don't regulate this but do care deeply.” 
Balen et al also comment that we highlight the intervention “Intralipid”, which they suggest is not widely used. This suggestion is not in keeping with our systematic findings. Our analysis stemmed from identification of claims of benefit on fertility centre websites, and we found three such instances claiming benefits for Intralipid; however, Intralipid is offered more widely (four offer Intralipid without stating a claim benefit on the website), meaning approximately 1 in 10 of the clinics we searched offer this intervention. We highlighted our concern for the widespread advocacy of Intralipid, especially given the lack of evidence for this intervention, as noted by both the BFS and Royal College of Obstetricians and Gynaecologists. 
Balen et al state that ‘many of the items have a clearly defined role in specific situations; e.g. for a man with a physical blockage sperm has to be extracted surgically, frozen and used in an IVF cycle combined with intra-cytoplasmic sperm injection (ICSI).’ We agree and ICSI was not categorised as an ‘add-on’ treatment in our analysis; we show to whom these interventions apply according to NICE guidance in our data supplement.  Balen et al say that looking for improved outcomes is irrelevant for this group of patients because ‘without those interventions (which are clearly not “add-ons”) IVF cannot happen.’ NICE, however, in their research recommendations (45 guideline CG156) state the need to ‘determine the long term effects of IVF with or without ICSI in children in the UK.’ 
We were pleased to note that Balen et al recognise that the evidence for some interventions is poor. We note that since the publication of our paper the evidence continues to evolve: a new systematic review on time-lapse imaging reported that there is insufficient evidence to support time-lapse imaging compared to conventional methods and there is a need for ‘more well-designed RCTs’  ; a further systematic review, published in May 2017, on strategies to improve fertilisation rates with assisted conception, further stated that ‘adequately powered multicentre randomised trials are required to evaluate these techniques before considering clinical application.’ 
We note that Balen et al have said there are shortcomings in our assessments of the quality of evidence for these interventions improving patients’ chances. If the authors can now provide these, we will happily review these and engage in constructive discussion on the emerging findings.
Finally, the use of ad hominem comments provides little benefit to anyone in this debate. Although we recognise the sensitivities and emotions that our work may have evoked amongst the IVF community, and despite widespread concerns over conflicts of interests in the IVF field, we would urge Balen et al to refocus their concerns at the IVF industry itself, and take a lead in utilising their extensive resources to reassure end users, the HFEA and the wider scientific community that their industry’s work is genuinely patient focused and evidence based. We would further suggest that the first step towards this would be to engage with the media, patient groups, policymakers and regulators to address the systemic findings that many interventions that lack good evidence to back them up are being offered to patients.
Carl Heneghan, Ben Goldacre, Elizabeth A Spencer, Jack O’Sullivan, Kamal R Mahtani,
 Spencer EA, Mahtani KR, Goldacre B, et al Claims for fertility interventions: a systematic assessment of statements on UK fertility centre websites. BMJ Open 2016;6:e013940. doi: 10.1136/bmjopen-2016-013940.
 Heneghan Carl, Spencer E A, Bobrovitz N, Collins D R J, Nunan D, Plüddemann A et al. Lack of evidence for interventions offered in UK fertility centres BMJ 2016; 355 :i6295
 Don’t dismiss conflict-of-interest concerns in IVF, they have a basis
 Carl Heneghan, Ben Goldacre, Kamal R. Mahtani. Why clinical trial outcomes fail to translate into benefits for patients. Trials, 2017, Volume 18, Number 1, Page 1
 Kurt T. Barnhart. Live Birth is the Correct Outcome for Clinical Trials Evaluating Therapy for the Infertile Couple Fertil Steril. 2014 May; 101(5): 1205–1208.
 HFEA: Innovation in regulation Feb 2017. http://ifqtesting.blob.core.windows.net/umbraco-website/1796/innovation-...
 Fertility problems: assessment and treatment. Clinical guideline [CG156] Published date: February 2013 Last updated: August 2016 https://www.nice.org.uk/guidance/cg156
 Does time-lapse imaging have favorable results for embryo incubation and selection compared with conventional methods in clinical in vitro fertilization? A meta-analysis and systematic review of randomized controlled trials. Chen M, Wei S, Hu J, Yuan J, Liu F.PLoS One. 2017 Jun 1;12(6):e0178720.
 Strategies to improve fertilisation rates with assisted conception: a systematic review. Jeve YB, Potdar N, Blower JA, Gelbaya T. Hum Fertil (Camb). 2017 May 26:1-19. doi: 10.1080/14647273.2017.1324182.
Competing interests: We have read and understood BMJ policy on declaration of interests and declare the following: CH has received grant funding from WHO, the National Institute for Health Research (NIHR), and the NIHR School of Primary Care and occasionally receives expenses for teaching evidence based medicine. He jointly runs the EvidenceLive Conference with BMJ, is editor in chief of the BMJ Evidence-Based Medicine Journal and receives payment as an NHS GP in urgent care. BG has received research funding from the Laura and John Arnold Foundation, the Wellcome Trust, the NIHR, the Health Foundation, and WHO; he also receives personal income from speaking and writing for lay audiences on the misuse of science. KM has received funding from the NIHR and the RCGP for independent research projects. The views expressed here are those of the authors and not necessarily those of any of the affiliated institutions mentioned in the manuscript.
14 June 2017
Professor of EBM
Ben Goldacre, Elizabeth A Spencer, Jack O'Sullivan, Kamal R Mahtani
Centre For Evidence-Based Medicine, Nuffield Department of Primary Care Health Scienes, University of Oxford
Radcliffe Observatory Quarter, Woodstock Road, Oxford. OX2 6GG
Competing interests: Competing interests: We have read and understood BMJ policy on declaration of interests and declare the following: CH has received grant funding from WHO, the National Institute for Health Research (NIHR), and the NIHR School of Primary Care and occasionally receives expenses for teaching evidence based medicine. He jointly runs the EvidenceLive Conference with BMJ, is Editor in Chief of the BMJ Evidence-Based Medicine Journal and receives payment as an NHS GP in urgent care. BG has received research funding from the Laura and John Arnold Foundation, the Wellcome Trust, the NIHR, the Health Foundation, and WHO; he also receives personal income from speaking and writing for lay audiences on the misuse of science. KM has received funding from the NIHR and the RCGP for independent research projects. The views expressed here are those of the authors and not necessarily those of any of the affiliated institutions mentioned in the manuscript.