Exploring changes over time and characteristics associated with data retrieval across individual participant data meta-analyses: systematic reviewBMJ 2017; 357 doi: https://doi.org/10.1136/bmj.j1390 (Published 05 April 2017) Cite this as: BMJ 2017;357:j1390
All rapid responses
The authors should be congratulated on correctly identifying half of the problem with most published meta-analyses. There are few meta-analyses that use even some individual patient data (IPD). However, I suggest rewording the conclusion of the abstract from "only 25% of published IPD meta-analyses have had access to all IPD" to "only 25% of published IPD meta-analyses have had access to some IPD and none had access to all IPD." What is provided in data sets for meta-analyses, data sharing web sites, and regulatory authorities is not all IPD but highly massaged and potentially manipulated data. Even the submissions to the U.S. FDA, for which the submitters have strong motivation to appear cooperative and FDA reviewers can be aggressive in requesting missing data, are incomplete. For a good discussion of the problems with IPD datasets see the FDA reviews such as the ones available at https://www.accessdata.fda.gov/drugsatfda_docs/nda/2011/022433Orig1s000M.... While all of the 640 pages are informative, there is a particularly relevant discussion of problems with IPD beginning at p. 46. It concludes with the following statement: ""At this time after thousands of man-hours of review and 88 NDA submissions I believe that we still have incomplete CRFs, incomplete data sets and audit trails, and only a few hundred scanty safety reports." I doubt that most published IPD meta-analyses entailed spending thousands of hours trying to get complete data for one study--or had the leverage of the FDA in trying to obtain such data.
On the other hand, other FDA reviews show the value of IPD meta-analyses having more complete data. Compare the IPD meta-analysis based on more complete data available at https://www.accessdata.fda.gov/drugsatfda_docs/nda/2011/022433Orig1s000M... (starting at p. 53) to published meta-analyses (and observational studies) regarding whether angiotensin receptor blockers are associated with an increased risk of cancer. From the careful FDA IPD meta-analysis one should be led to a different appreciation of the risk of lung cancer with these drugs. I agree with the authors that "IPD meta-analyses are considered to be the 'gold standard' for the synthesis of data from clinical research studies."
Competing interests: No competing interests