PLoS ONE plos plosone PLoS ONE 1932-6203 Public Library of Science San Francisco, USA PONE-D-14-04066 10.1371/journal.pone.0102670 Research ArticleMedicine and health sciencesClinical medicineClinical trialsPhase III clinical investigationPhysical sciencesMathematicsStatistics (mathematics)Statistical methodsMeta-analysisResearch and analysis methodsResearch assessmentSystematic reviewsResearch designClinical research design Accumulating Research: A Systematic Account of How Cumulative Meta-Analyses Would Have Provided Knowledge, Improved Health, Reduced Harm and Saved Resources Cumulative Meta-Analyses ClarkeMike1* BriceAnne2 ChalmersIain2 All-Ireland Hub for Trials Methodology Research, Queen's University Belfast, Belfast, United Kingdom James Lind Initiative, Oxford, United Kingdom GluudLise Lotte Editor Copenhagen University Hospital Gentofte, Denmark * E-mail: m.clarke@qub.ac.uk

The authors have declared that no competing interests exist.

Conceived and designed the experiments: MC AB IC. Performed the experiments: MC AB IC. Analyzed the data: MC AB IC. Wrote the paper: MC AB IC.

 2014 2872014 9 7 e102670 632014 2062014 2014 Clarke et alThis is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Background

“Cumulative meta-analysis” describes a statistical procedure to calculate, retrospectively, summary estimates from the results of similar trials every time the results of a further trial in the series had become available. In the early 1990s, comparisons of cumulative meta-analyses of treatments for myocardial infarction with advice promulgated through medical textbooks showed that research had continued long after robust estimates of treatment effects had accumulated, and that medical textbooks had overlooked strong, existing evidence from trials. Cumulative meta-analyses have subsequently been used to assess what could have been known had new studies been informed by systematic reviews of relevant existing evidence and how waste might have been reduced.

 Methods and Findings

We used a systematic approach to identify and summarise the findings of cumulative meta-analyses of studies of the effects of clinical interventions, published from 1992 to 2012. Searches were done of PubMed, MEDLINE, EMBASE, the Cochrane Methodology Register and Science Citation Index. A total of 50 eligible reports were identified, including more than 1,500 cumulative meta-analyses. A variety of themes are illustrated with specific examples. The studies showed that initially positive results became null or negative in meta-analyses as more trials were done; that early null or negative results were over-turned; that stable results (beneficial, harmful and neutral) would have been seen had a meta-analysis been done before the new trial; and that additional trials had been much too small to resolve the remaining uncertainties.

 Conclusions

This large, unique collection of cumulative meta-analyses highlights how a review of the existing evidence might have helped researchers, practitioners, patients and funders make more informed decisions and choices about new trials over decades of research. This would have led to earlier uptake of effective interventions in practice, less exposure of trial participants to less effective treatments, and reduced waste resulting from unjustified research.

 The authors have no support or funding to report. Introduction

In 1992, a team lead by Tom Chalmers and Fred Mosteller introduced the term ‘cumulative meta-analysis' to describe a statistical procedure to calculate, retrospectively, summary estimates based on the results of similar trials every time the results of a further trial in the series had become available [1]. One of their two papers published in 1992 made clear how important this procedure was for auditing both research and healthcare advice. Comparisons of the results of cumulative meta-analyses of treatments for myocardial infarction with the advice that had been promulgated through medical textbooks [2] made clear not only that research had continued long after robust estimates of treatment effects had accumulated, but also that medical textbooks had overlooked strong, existing evidence from clinical trials, both of beneficial and of lethal effects of treatments [3].

Cumulative meta-analyses have subsequently been used to assess what could have been known had the design of new studies been informed by reference to systematic reviews of relevant existing evidence and how these might have reduced waste [4]. Cumulative meta-analyses emphasise the need for the design of new studies to be informed by existing research [5] and for the results of new studies to be set in the context of updated systematic reviews of the relevant evidence from all sufficiently similar studies [6]. The idea of using the accumulating evidence to make decisions about the design and ongoing conduct of trials is not new: the report of a case study published by Henderson and colleagues nearly 20 years ago noted “Our thesis is that if related published trials are available, a meta-analysis should be started in the planning stages of a clinical trial, continued through the ongoing conduct of the trial, and performed as one analysis among many in the final analysis of the trial” [7]. Such reviews and meta-analyses help to provide the ethical, scientific and environmental justification for both new study and for any future studies [8].

In this methodological review, we use systematic methods to search for and summarise the findings of cumulative meta-analyses of studies of the effects of clinical interventions, published from 1992 to 2012. We describe the different settings for these studies and explore their findings in the context of unnecessary duplication of effort or waste if trials were done after a robust finding would have been found if a review and meta-analysis of existing research had been performed. By conducting this research as a systematic review, our aim is to provide the most comprehensive collection of cumulative meta-analysis of studies of healthcare interventions. The searching for this review also identified several cumulative meta-analyses in other types of health research, which are not summarised here but have been discussed in brief elsewhere [4]. For example, if epidemiological studies investigating possible aetiological factors in sudden infant death syndrome had taken proper account of the accumulating evidence, the lethal effect of ‘front lying’ would have been recognized at least a decade earlier, and tens of thousands of infant deaths could have been avoided [9]. A cumulative meta-analysis of 55 studies that continued to be conducted over more than two decades showed that for over 17 years there had been ample evidence that never-smoking women who had been exposed to spousal smoking were more likely than controls to develop lung cancer [10].

 Methods Inclusion criteria

Studies were eligible if they included a cumulative meta-analysis of studies of the relative effects of alternative healthcare interventions. Ideally, the cumulative meta-analysis would be presented as a graph showing the summary estimates as each study's result was added to the meta-analyses in the order this had been published or became available in some other way. However, studies were eligible if these graphs or the separate summary estimates were not available, but the general findings or implications of the cumulative meta-analyses were reported. Studies were eligible if a single cumulative meta-analysis was presented, or if a meta-epidemiological project had been done in which numerous systematic reviews or meta-analyses were included. Eligible cumulative meta-analyses had sometimes been done as part of the planning process for a new study to explore how the evidence base had evolved. Studies were not included if only surrogate outcome measures were used unless - like blood pressure and severe anaemia, for example - these were unambiguously important. The searches also identified cumulative meta-analysis in other types of health research (including observational epidemiology and genetics) but these were not eligible for this review.

 Search strategy

We wished to identify published reports containing graphs of cumulative meta-analyses. An initial search of Pubmed Clinical Queries using the term ‘cumulative meta-analysis’ retrieved 822 records. The selected papers were screened by one author (IC), who selected a set of 23 records which were added to a set of core articles that had been previously identified as part of this author's general interest in this area. This full set of papers were then analysed to obtain ideas for free-text search terms and index or MeSH terms for use in a final search strategy.

Searches were then run on the following databases: MEDLINE, OVID (1946–2012, In-Process and other non-indexed citations); EMBASE, OVID (1947–2012); the Cochrane Methodology Register (2012); and Science Citation Index (2012). No language or publication date restrictions were applied. The following search strategy was used for MEDLINE: cumulative adj10 meta?analys$.mp. OR (cumulative.mp. and meta-analysis.pt.). The search of the Cochrane Methodology Register combined the output of a simple search for the word ‘cumulative’ in the abstract with that for a search for records that had been assigned the term ‘Meta-analysis updating and cumulative meta-analysis’ by the compilers of the Register. The searches were conducted in June 2012.

The reference lists of relevant systematic reviews and other publications were checked to identify additional articles [11]. A search was undertaken in Google Scholar to identify relevant terms in the full text of articles where relevant concepts had not been located in bibliographic databases using searches of titles, abstracts and index terms.

 Assessment of studies, data extraction and analysis

Records were exported from Endnote to Sente for first screening by one of the authors (AB). This screening was deliberately over-inclusive, and all potentially eligible articles or unclear articles were checked by a second author (IC). The eligibility of all studies arising from this process was then confirmed by a third author (MC) before their inclusion in this review.

The process of data extraction was piloted in ten records and emerging themes identified. A final data extraction form was then agreed. All eligible papers were read, and passages of text commenting on the details of the cumulative meta-analysis and the implications of the completed cumulative meta-analyses were identified and extracted. This was done by one author (AB) and the data extracted were checked against the original articles by the other authors (IC and MC).

No attempts were made to combine the results of the included studies, because the aim of this review is to present a range of examples and identify themes across the cumulative meta-analyses identified, rather than to attempt to generate a statistical finding. Similarly, this review does not seek to estimate the incidence or prevalence of trials that have been conducted unnecessarily, since that would be done better with a comprehensive study of large cohorts of trials addressing the same or similar questions. Instead, it seeks to identify and report a range of examples to examine whether cumulative meta-analyses might have led to different choices in the conduct of new research.

 Results Results of the search

This initial screening identified 942 records needing inspection, of which 818 were excluded in the initial screening. This screening identified 46 records for inclusion and 78 needing further review. The second screening step excluded 74 of these, leaving 50 reports, including more than 1500 cumulative meta-analyses of clinical intervention studies, as eligible.

 Included studies

The included studies were published from 1992 to 2012 inclusive. Most of these studies focused on a single question for a systematic review or meta-analysis, and included just one graphical presentation of a cumulative meta-analysis. The 50 studies addressed a wide range of health problems. Fourteen covered aspects of surgery for a variety of problems (reducing infection and bleeding and improving surgical technique). Eleven studies concerned heart disease, and there were at least two for each of stroke, neonatal problems, infectious disease, cancer, mental health, dental health and respiratory illness. There were single studies in obstetrics, musculoskeletal disease, kidney disease and gastrointestinal disease. The number of individual research studies in each of the cumulative meta-analyses we identified ranged from 4 trials of continuous versus interrupted techniques for elective midline abdominal fascial closure [12] to more than 60 trials for aprotinin in cardiac surgery [13].

Nearly all of the cumulative meta-analyses had been conducted as retrospective exercises to explore how the evidence base had evolved. However, some were used to inform the design of a new study. For example, Algra and van Gijn presented a cumulative meta-analyses which was used to inform the need for the European and Australian Stroke Prevention in Reversible Ischaemia Trial (ESPRIT) [14].

A variety of themes were apparent from the cumulative meta-analyses and these are illustrated here with specific examples. A detailed explanation is not provided for each cumulative meta-analyses, but the citations for all the included studies are available in Appendix S1. The included studies showed that initially positive results became null or negative in the meta-analysis as more trials were done; that early null or negative results were over-turned; that stable results (beneficial, harmful and neutral) would have been seen had a meta-analysis of existing evidence been done before the new trial; and that additional trials had been much too small to resolve the remaining uncertainties.

 Positive results becoming less so

Cumulative meta-analyses have shown how replications have challenged initially favourable results [15]–[18] where the early trials were favourable but not statistically significant. For example, Klein and colleagues examined the effects of cognitive behavioural therapy (CBT) for adolescent depression, using the publication dates for 11 randomised trials to build a cumulative meta-analysis. They found that the effect size from the meta-analyses had decreased steadily from the large effects observed in the earliest trials, with narrowing confidence intervals as the data accumulated. However, they also noted that the trend could be related to methodological differences between the studies and wrote “these differences appear to reflect both a shift from an initial emphasis on demonstrating the efficacy of treatment in controlled research settings to an emphasis on demonstrating the effectiveness of treatment and the application of increased statistical and methodological rigor over time”. They concluded that “the results indicate that CBT may be effective for the acute treatment of depression among adolescents, although treatment effects may be more modest in clinical settings than findings from early trials would suggest” [17].

 Null or negative results becoming positive

Cumulative meta-analyses have also shown how replications have sometimes challenged initially unfavourable results [19]–[20], but these examples are fewer and weaker than for the previous theme. For example, in a systematic review comparing plating versus intramedullary nailing of humeral shaft fractures in adults, the cumulative meta-analyses by Li and colleagues showed that intramedullary nailing might increase the re-operation rate in studies conducted before 2000 (odds radio (OR): 0.39, 95% confidence interval (CI): 0.17 to 0.90, P = 0.03). They noted that although the point estimate was still unfavourable, the difference was not statistically significant in studies conducted after that year (OR: 0.54, 95% CI: 0.27 to 1.08, P = 0.08) [19].

 Stable results

This collection of cumulative meta-analyses includes many that show that a systematic review of existing research would have reduced uncertainty about an intervention. They showed how morbidity and mortality might have been reduced, both for patients in general and for participants in trials in which they may have continued to be allocated to a placebo or a control group when the intervention could have been shown to be effective in a meta-analysis [1]–[2], [12]–[14], [18], [21]–[38].This observation also has implications for waste in research, since new studies might have been regarded as unjustified if a systematic review and meta-analysis had been done as part of its design phase. Two examples, through two decades, are used to illustrate this.

In 1992, Lau and colleagues reported a collection of cumulative meta-analyses of clinical trials that evaluated 15 treatments and preventive measures for acute myocardial infarction. They found a consistent, statistically significant reduction in total mortality for the use of streptokinase (OR: 0.74, 95% CI: 0.59 to 0.92) would have been shown in 1973, after only eight trials involving 2432 patients had been completed. The results of 25 subsequent trials, which enrolled an additional 34,542 patients through 1988, had little or no effect on the odds ratio establishing efficacy, but narrowed the 95% confidence interval [1].

The cumulative meta-analysis by Fergusson and colleagues of aprotinin in cardiac surgery found a dramatically beneficial result in the first trial of 22 patients in 1987 (OR for perioperative blood transfusion: 0.03, 95% CI: 0.00 to 0.56), with the point estimate declining over subsequent randomised trials. By the twelfth study (published in June 1992), the cumulative effect estimate for the odds ratio had stabilized (OR: 0.25, P<0.000001) with a total of approximately 2500 patients randomised. Throughout the cumulative meta-analysis, the upper limit of the confidence interval did not go higher than 0.65, and in this study, the final meta-analyses was 0.34 (95% CI: 0.24–0.41) after the publication of a trial in June 2002, by which time a total of more than 8000 patients had been randomised [13].

 New trials too small to resolve remaining uncertainties

Although systematic reviews of existing evidence sometimes reveal data that are sufficient to answer research questions, cumulative meta-analyses, like that of tranexamic acid [39], have also exposed questions that remain unanswered, but continue to be addressed in studies that are much too small to resolve the remaining uncertainties [15], [40]–[56]. The importance of addressing uncertainties was revealed when a systematic review showed that a series of 16 under-powered trials conducted over 23 years failed to indicate whether steroids given to patients with traumatic brain injury reduced or increased their chances of survival. The uncertainty was only resolved when a very large trial showed reliably that the treatment increased mortality [57].

 Discussion

This large collection of cumulative meta-analyses, from across health and social care, and covering a wide range of research into the effects of interventions, highlights how a review of the existing evidence might have helped researchers, practitioners, patients and funders make more informed decisions and choices about new clinical trials over many decades of research. The importance of taking account of earlier research is not a new concept for either the design or interpretation of new studies. For example, it was highlighted to the British Association for the Advancement of Science by Lord Rayleigh in the nineteenth century [58]–[59].

The cumulative meta-analyses that have been considered together for the first time in this review provide cautionary tales about how new research might have been designed and implemented, when the existing research would have shown it to be unnecessary. Other research, taking a different approach to this review, has also highlighted this problem. In 2011, Herbison and colleagues reported cumulative meta-analysis of data from 65 meta-analyses from 18 Cochrane Reviews. They found that it took a median of four (interquartile range: 1 to 6) studies to get within 10% of the final point estimate. However, they noted that although their study suggests that, in many cases, only a few trials are necessary before getting a reasonably robust answer, it is difficult to know which meta-analyses will change further and which will not. They conclude “it is still unclear what characteristics of early trials will lead to more confidence being placed in the results of individual meta-analyses” [35].

It is also important to recognise how unreliable initial evidence can be [60]–[61]. An analysis of data from 85,000 meta-analyses with binary outcomes in Cochrane Reviews showed that early trials tend to overestimate treatment effects [62]. This may result, for example, from studies selecting unrepresentative subgroups of patients known to have responded favourably to similar interventions previously, or from excluding patients who have not responded [63]; from biased under-reporting of early trials with disappointing results [64]; or time-lag bias in the publication of the results of trials [65]. As a result, early studies and meta-analyses of them may tend to yield inflated estimates of effects and this needs to be taken into account in considering proposals for additional studies [66]. However, an assessment of the existing evidence is still crucial to providing the ethical, scientific and environmental justification for proposed new trials. Without such reviews, those responsible for proposed new trials cannot make a well-informed decision about whether to proceed with them, and they need to include careful assessment of the quality and risk of bias in the studies being brought together [67]. This should influence the interpretation of the results of the cumulative meta-analysis, alongside consideration of the possibility that spurious results might arise due to cumulative testing. This may require the use of statistical techniques such as sequential analysis, and the need to consider the statement “Don't Ignore Chance Effects” when building from early, positive findings [68].

In considering possible limitations of our research, we note that systematic reviews of healthcare interventions are subject to the impact of selective reporting by researchers, in which whole studies might not be published [69] or, even if studies are published, particular findings might be excluded because of the authors' or editors' opinions about the findings [70]. This methodological review could also be subject to such biases, where cumulative meta-analyses may have been performed but not reported. In contrast to clinical trials [71] or, more recently, systematic reviews with health outcomes [72], there is no widely available system to register prospectively methodological research, such as cumulative meta-analysis. This makes it impossible to determine the extent of selective reporting of cumulative meta-analysis, or its potential impact on our conclusions. Furthermore, although there were relatively few examples of cumulative meta-analyses in which benefits appeared for an intervention after the initial trials had shown null or negative results, this is not surprising, because awareness of such early results, even without a formal meta-analysis, might discourage future research. By contrast, while early positive results might lead to the conduct of new trials to confirm those results or to test their reproducibility in other settings [73].

Despite these limitations, however, and given that this review found such a breadth of examples, it is likely that our general finding is likely to be valid, namely, that there is a substantial problem of waste in research resulting from unnecessary duplication because existing research has not been reviewed before and after new studies are done [4].

 Conclusions

This analysis of 50 reports including over 1500 cumulative meta-analyses of clinical intervention studies shows that, had researchers assessed systematically what was already known, some beneficial and harmful effects of treatments could have been identified earlier and might have prevented the conduct of the new trials. This would have led to the earlier uptake of effective health and social care interventions in practice, less exposure of trial participants to less effective treatments, and reduced waste resulting from unjustified research.

We do not argue that the conduct of a new trial in the presence of apparent certainty from a meta-analysis of existing research is necessarily wrong in all circumstances. The new trial might serve to fill a gap or resolve a remaining uncertainty for particular types of intervention, patient or setting, or seek to assess important outcomes that were not measured in the earlier trials [74]. However, we do argue that people designing, funding, conducting and then interpreting new studies should make their decision to do so in light of an up-to-date systematic review and, if possible, meta-analysis of existing related research. This would allow them to show the ethical, scientific and environmental basis for their proposed new trials, and to demonstrate that they would not be wasteful and justify their decision [8].

 Supporting Information

Cumulative meta-analyses of studies of the effects of healthcare interventions.

(DOCX)

 References LauJ, AntmanEM, Jimenez-SilvaJ, MostellerF, ChalmersTC (1992) Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine 327: 248–254. AntmanEM, LauJ, KupelnickB, MostellerF, ChalmersTC (1992) A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 268: 240–248. Rennie D, Chalmers I (2009) Exposing the dangers to patients of medical reviews and textbooks that ignore scientific principles. JLL Bulletin: Commentaries on the history of treatment evaluation (www.jameslindlibrary.org). ChalmersI, BrackenMB, DjulbegovicB, GarattiniS, GrantJ, et al. (2014) How to increase value and reduce waste when research priorities are set. Lancet 383: 156–165. JonesAP, ConroyE, WilliamsonPR, ClarkeM, GambleC (2013) The use of systematic reviews in the planning, design and conduct of randomised trials: a retrospective cohort of NIHR HTA funded trials. BMC Medical Research Methodology 13: 50. ClarkeM, HopewellS, ChalmersI (2010) Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet 376: 20–21. HendersonWG, MoritzT, GoldmanS, CopelandJ, SethiG (1995) Use of cumulative meta-analysis in the design, monitoring, and final analysis of a clinical trial: a case study. Controlled Clinical Trials 16: 331–341. Sustainable Trials Study Group (2007) Towards sustainable clinical trials. BMJ 334: 671–673. GilbertR, SalantiG, HardenM, SeeS (2005) Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. International Journal of Epidemiology 34: 874–887. TaylorR, NajafiF, DobsonA (2007) Meta-analysis of studies of passive smoking and lung cancer: Effects of study type and continent. International Journal of Epidemiology 36: 1048–1059. HorsleyT, DingwallO, SampsonM (2011) Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews (8): MR000026. DienerMK, VossS, JensenK, BuchlerMW, SeilerCM (2010) Elective midline laparotomy closure: the INLINE systematic review and meta-analysis. Annals of Surgery 251: 843–856. FergussonD, GlassKC, HuttonB, ShapiroS (2005) Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clinical Trials 2: 218–229. AlgraA, van GijnJ (1999) Cumulative meta-analysis of aspirin efficacy after cerebral ischaemia of arterial origin. Journal of Neurology, Neurosurgery and Psychiatry 66: 255. BollenCW, UiterwaalCS, van VughtAJ (2003) Cumulative metaanalysis of high-frequency versus conventional ventilation in premature neonates. American Journal of Respiratory and Critical Care Medicine 168: 1150–1155. HansonRK, BroomI (2005) The utility of cumulative meta-analysis: application to programs for reducing sexual violence. Sexual Abuse: Journal of Research & Treatment 17: 357–373. KleinJB, JacobsRH, ReineckeMA (2007) Cognitive-behavioral therapy for adolescent depression: a meta-analytic investigation of changes in effect-size estimates. Journal of the American Academy of Child and Adolescent Psychiatry 46: 1403–1413. ZhangW, NukiG, MoskowitzRW, AbramsonS, AltmanRD, et al. (2010) OARSI recommendations for the management of hip and knee osteoarthritis, part iii: changes in evidence following systematic cumulative update of research published through January 2009. Osteoarthritis and Cartilage 18: 476–499. LiLH, SunTS, LiuZ, GuoYZ, LiSG, et al. (2008) Plating versus intramedullary nailing of humeral shaft fractures in adults: a systematic review. Chinese Journal of Evidence-Based Medicine 8: 662–667. BhandariM, SchemitschE, JonssonA, ZlowodzkiM, HaidukewychGJ (2009) Gamma nails revisited: gamma nails versus compression hip screws in the management of intertrochanteric fractures of the hip: a meta-analysis. Journal of Orthopaedic Trauma 23: 460–464. MittendorfR, AronsonMP, BerryRE, WilliamsMA, KupelnickB, et al. (1993) Avoiding serious infections associated with abdominal hysterectomy: a meta-analysis of antibiotic prophylaxis. American Journal of Obstetrics and Gynecology 169: 1119–1124. CarpinelliL, PrimignaniM, BianchiM, CivettiniF, De FranchisR (1995) Treatment of bleeding oesophageal varices: a meta-analysis. Argomenti di Gastroenterologia Clinica 8: 5–15. LauJ, SchmidCH, ChalmersTC (1995) Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology 48: 45–57. SinclairJC (1995) Meta-analysis of randomized controlled trials of antenatal corticosteroid for the prevention of respiratory distress syndrome: discussion. American Journal of Obstetrics and Gynecology 173: 335–344. WhitingGW, LauJ, KupelnickB, ChalmersTC (1995) Trends in inflammatory bowel disease therapy: a meta-analytic approach. Canadian Journal of Gastroenterology 9: 405–411. ClarkO, AdamsJR, BennettCL, DjulbegovicB (2002) Erythropoietin, uncertainty principle and cancer related anaemia. BMC Cancer 2: 23. GuptaAK, RyderJE, JohnsonAM (2004) Cumulative meta-analysis of systemic antifungal agents for the treatment of onychomycosis. British Journal of Dermatology 150: 537–544. JuniP, NarteyL, ReichenbachS, SterchiR, DieppePA, et al. (2004) Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet 364: 2021–2029. CoomarasamyA, ThangaratinamS, GeeH, KhanKS (2006) Progesterone for the prevention of preterm birth: a critical evaluation of evidence. European Journal of Obstetrics, Gynecology, & Reproductive Biology 129: 111–118. GilbodyS, BowerP, FletcherJ, RichardsD, SuttonAJ (2006) Collaborative care for depression: a cumulative meta-analysis and review of longer-term outcomes. Archives of Internal Medicine 166: 2314–2321. WallaceAE, NeilyJ, WeeksWB, FriedmanMJ (2006) A cumulative meta-analysis of selective serotonin reuptake inhibitors in pediatric depression: did unpublished studies influence the efficacy/safety debate? Journal of Child and Adolescent Psychopharmacology 16: 37–58. PoolmanRW, FarrokhyarF, BhandariM (2007) Hamstring tendon autograft better than bone patellar-tendon bone autograft in ACL reconstruction: a cumulative meta-analysis and clinically relevant sensitivity analysis applied to a previously published analysis. Acta Orthopaedica 78: 350–354. RossJS, MadiganD, HillKP, EgilmanDS, WangY, et al. (2009) Pooled analysis of rofecoxib placebo-controlled clinical trial data: lessons for postmarket pharmaceutical safety surveillance. Archives of Internal Medicine 169: 1976–1985. DeshpandeG, RaoS, PatoleS, BulsaraM (2010) Trial sequential analysis of updated meta-analysis of probiotics for preventing necrotising enterocolitis in preterm VLBW neonates: are the results conclusive? Journal of Paediatrics and Child Health, Conference: 14th Annual Congress of the Perinatal Society of Australia and New Zealand, PSANZ 2010 Wellington New Zealand. 46: 65. HerbisonP, Hay-SmithJ, GillespieWJ (2011) Meta-analyses of small numbers of trials often agree with longer-term results. Journal of Clinical Epidemiology 64: 145–153. ShethU, SimunovicN, TornettaP3rd, EinhornTA, BhandariM (2011) Poor citation of prior evidence in hip fracture trials. Journal of Bone and Joint Surgery - Series A 93: 2079–2086. WellsG, ParkashR, HealeyJS, TalajicM, ArnoldJM, et al. (2011) Cardiac resynchronization therapy: a meta-analysis of randomized controlled trials. Canadian Medical Association Journal 183: 421–429. MillsEJ, WuP, AlbertonM, KantersS, LanasA, et al. (2012) Low-dose aspirin and cancer mortality: a meta-analysis of randomized trials. American Journal of Medicine 125: 560–567. KerK, EdwardsP, PerelP, ShakurH, RobertsI (2012) Effect of tranexamic acid on surgical bleeding: systematic review and cumulative meta-analysis. BMJ 344: e3054. PeterJV, MoranJL, Phillips-HughesJ, WarnD (2002) Noninvasive ventilation in acute respiratory failure - a meta-analysis update. Critical Care Medicine 30: 555–562. WardlawJM, SandercockPAG, BergeE (2003) Thrombolytic therapy with recombinant tissue plasminogen activator for acute ischemic stroke: where do we go from here? A cumulative meta-analysis. Stroke 34: 1437–1442. WeigelR, SchmiedekP, KraussJK (2003) Outcome of contemporary surgery for chronic subdural haematoma: evidence based review. Journal of Neurology, Neurosurgery & Psychiatry 74: 937–943. KwakkelG, Van PeppenR, WagenaarRC, DauphineeSW, RichardsC, et al. (2004) Effects of augmented exercise therapy time after stroke: a meta-analysis. Stroke 35: 2529–2536. TrikalinosTA, ChurchillR, FerriM, LeuchtS, TuunainenA, et al. (2004) Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. Journal of Clinical Epidemiology 57: 1124–1130. DevereauxPJ, BeattieWS, Choi,PT, BadnerNH, GuyattGH, et al. (2005) How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. British Medical Journal 331: 313–316. MatchabaP, GittonX, KrammerG, EhrsamE, SloanVS, et al. (2005) Cardiovascular safety of lumiracoxib: a meta-analysis of all randomized controlled trials ≥1 week and up to 1 year in duration of patients with osteoarthritis and rheumatoid arthritis. Clinical Therapeutics 27: 1196–1214. MolesDR, NeedlemanIG, NiedermanR, LauJ (2005) Introduction to cumulative meta-analysis in dentistry: lessons learned from undertaking a cumulative meta-analysis in periodontology. Journal of Dental Research 84: 345–349. ThomeU, CarloWA, PohlandtF (2005) Ventilation strategies and outcome in randomised trials of high frequency ventilation. Archives of Disease in Childhood: Fetal and Neonatal Edition 90: F466–F473. PuhanMA, VollenweiderD, SteurerJ, BossuytPM, Ter RietG (2008) Where is the supporting evidence for treating mild to moderate chronic obstructive pulmonary disease exacerbations with antibiotics? A systematic review. BMC Medicine 6: 28. BrokJ, GluudLL, GluudC (2010) Meta-analysis: ribavirin plus interferon vs. Interferon monotherapy for chronic hepatitis C - an updated Cochrane review. Alimentary Pharmacology and Therapeutics 32: 840–850. KeusF, WetterslevJ, GluudC, GooszenHG, van LaarhovenCJHM (2010) Trial sequential analyses of meta-analyses of complications in laparoscopic vs. small-incision cholecystectomy: more randomized patients are needed. Journal of Clinical Epidemiology 63: 246–256. OhSW, MyungSK, ParkJY, LymYL, JuW (2010) Hormone therapy and risk of lung cancer: a meta-analysis. Journal of Women's Health 19: 279–288. PalmerS, NavaneethanS, CraigJ, JohnsonD, TonelliM, et al. (2010) Erythropoiesis-stimulating agents in people with chronic kidney disease: a cumulative meta-analysis. NDT Plus, Conference: 17th ERA-EDTA Congress - II DGfN Congress Munich Germany 3: iii80–iii81. DentL, TaylorR, JollyK, RafteryJ (2011) "Flogging dead horses": evaluating when have clinical trials achieved sufficiency and stability? A case study in cardiac rehabilitation. Trials 12: 83. BelseyJ, CrostaC, EpsteinO, FischbachW, LayerP, et al. (2012) Meta-analysis: the relative efficacy of oral bowel preparations for colonoscopy 1985–2010. Alimentary Pharmacology & Therapeutics 35: 222–237. MartelG, CrawfordA, BarkunJS, BousheyRP, RamsayCR, et al. (2012) Expert opinion on laparoscopic surgery for colorectal cancer parallels evidence from a cumulative meta-analysis of randomized controlled trials. PLoS ONE 7: e35292. RobertsI, YatesD, SandercockP, FarrellB, WasserbergJ, et al. (2004) Effect of intravenous corticosteroids on death within 14 days in 10008 adults with clinically significant head injury (MRC CRASH trial): randomised placebo-controlled trial. Lancet 364: 1321–1328. Lord Rayleigh (1885) Address by the Rt. Hon. Lord Rayleigh. Report of the fifty-fourth meeting of the British Association for the Advancement of Science; held at Montreal in August and September 1884 London: John Murray. 3–23. ClarkeM (2004) Doing New Research? Don't Forget the Old. PLoS Medicine 1: e35. RothwellPM, RobertsonG (1997) Meta-analyses of randomised controlled trials. Lancet 350: 1181–1182. Davey SmithG, EggerM (1997) Meta-analyses of randomised controlled trials. Lancet 350: 1182. PereiraTV, HorwitzRI, IoannidisJP (2012) Empirical evaluation of very large treatment effects of medical interventions. JAMA 308: 1676–1684. Rothwell PM, editor (2007) Treating individuals: from randomised trials to personalised medicine. London: Lancet. ChanA-W, SongF, VickersA, JeffersonT, DickersinK, et al. (2014) Increasing value and reducing waste: addressing inaccessible research. Lancet 383: 257–266. HopewellS, ClarkeM, StewartL, TierneyJ (2007) Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews (2): MR000011. PereiraTV, IoannidisJP (2011) Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology 64: 1060–1069. AdamsCE, PolzmacherS, WolffA (2013) Systematic reviews: what needs to be done and not to be done. Journal of Evidence-Based Medicine 6: 232–235. ClarkeM, HalseyJ (2014) Dicing with chance, life and death in systematic reviews and meta-analyses: D.I.C.E. 3, a simulation study. Journal of the Royal Society of Medicine 107: 116–119. HopewellS, LoudonK, ClarkeMJ, OxmanAD, DickersinK (2009) Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews (1): MR000006. KirkhamJJ, DwanKM, AltmanDG, GambleC, DoddS, et al. (2010) The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340: c365. GhersiD, PangT (2009) From Mexico to Mali: four years in the history of clinical trial registration. Journal of Evidence-Based Medicine 2: 1–7. BoothA, ClarkeM, DooleyG, GhersiD, MoherD, et al. (2013) PROSPERO at one year: an evaluation of its utility. Systematic Reviews 2: 4. Liver Infusion Meta-analysis Group (1997) Portal vein chemotherapy for colorectal cancer: a meta-analysis of 4000 patients in 10 studies. Journal of the National Cancer Institute 89: 497–505. BrownP, BrunnhuberK, ChalkidouK, ChalmersI, ClarkeM, et al. (2006) How to formulate research recommendations. BMJ 333: 804–806.