Large randomized clinical trials (RCTs) provide the best evidence to justify new treatments or to identify treatments that do not improve patient outcomes. Gordon and colleagues reported that most large NHLBI-funded trials produce null results[1], but their analysis only considered papers published after 2000. Considering all large trials over the last 40 years, we explore whether there has been a trend toward null finding in recent years and consider potential explanations for trends in observing null outcomes.

Among 4,089 individual years of grant funding, almost half were excluded as multiple years of the same grant and over 20% were excluded because they were single sites in multi-site trials, coordinating centers, or ancillary studies of the same trial. An additional 1,176 grant abstracts did not match our criteria and were excluded (see S1 Table for detailed reasons). Main outcome papers were searched for 84 trials; 10 were not published and 25 did not match search criteria and were excluded (See S1 Fig for the PRISMA diagram and S1 Table for the number of studies excluded by reason.) Following exclusions, we identified a total of 49 funded grants. Four of these grants resulted in multiple unique trials (ACCORD Blood Pressure, Diabetes, and Lipid; ALLHAT-BP, DOX, LLT; WHI Estrogen and Estrogen-Progestin, and WHS aspirin and vitamin E). A total of 55 trials were analyzed– 30 were published prior to 2000 and 25 were published in 2000 or later (see S2 Table for list of included trials. A complete list of the references also appears in the Supplemental Materials).

Fig 1 plots the relative risks of the primary outcome by the publication year of the main outcome paper. Because it was an extreme outlier, the CAST study is excluded from the figure. Prior to publication in 2000, studies often showed benefits of treatments with the notable exception of CAST (not shown in figure). Following 2000, confidence intervals for relative risk ratios included 1.0 in all cases, with the exceptions of the PREVENT and the SANDS trials (benefit) and the Women’s Health Initiative (Harm). In addition, the variability in RRs was considerably reduced after the year 2000 (Fig 2).

Results for all cause mortality were similar. Prior to 2000, 24 trials reported all cause-mortality and 5 reported significant reductions in total mortality (25%), 18 were null (71%) and one (CAST) reported significant harm (Table 3). Following the year 2000, no study showed a significant benefit for total mortality. An expanded presentation of the results in given in the online supplemental materials, including a figure summarizing results for all cause mortality.

We considered a variety of explanations for the trend toward null results that emerged around 2000. (Detailed tables are given in online supplemental materials). One possibility is that more recent trials may have evaluated their treatment drug against clinically effective alternatives instead of placebos. We do not find this suggestion likely because 60% of the large NHLBI trials published prior to 2000 used a placebo as the comparator in contrast to 64% trials published after 2000 (see S3 Table). Placebos were used as the comparator at about the same rate prior to and after the year 2000 (p = .979).

To investigate the effect of industry co-sponsorship, we tabulated sponsorship for all reports. Unfortunately, industry co-sponsorship was not always reported prior to the year 2000 and journals did not uniformly require disclosure. After the year 2000, when the International Committee of Medical Journal Editors (ICMJE) asked for disclosure, it became apparent that industry co-sponsorship is very common. In our sample, 23 of 25 (92%) of the NHLBI trials published after 2000 had partial industry sponsorship or contribution of medications. All but two of these trials obtained null results. We also looked at previous financial relationships between investigators and industry. Prior to 2000, these relationships were reported in only 1 of the 30 trials (3%). Even after 2000, 28% of the studies did not include a disclosure section. But among articles that included disclosures, there was a financial consulting relationship between at least one author and industry in all (100%) of the cases. Industry influence would produce a bias in favor of positive results, so connections between investigators and industry is not a likely explanation for the trend toward null results in recent years.

We considered a variety of aspects of transparent reporting. Prior to 2000, 5 of the 30 published trials (17%) included a diagram that clearly accounted for the number of participants at each phase of the project. Following 2000, publications were significantly more likely to account for patients throughout the study: 14 of the 25 trials (56%) included such a flow diagram (χ2 = 9.22, p = 0.002). After the year 2000 all of the published papers clearly identified primary outcome variable, while the primary outcome variable was not specified in 23% of the publications prior to 2000 (χ2 = 4.75, p = 0.03).

A final explanation for the trend toward null reports is that current authors face greater constraints in reporting the results of their studies. In our review, the year 2000 marks the beginning of a natural experiment. After the year 2000, all (100%) of large NHLBI were registered prospectively in ClinicalTrials.Gov prior to publication. Prior to 2000 none of the trials (0%) were prospectively registered. Although many of the earlier studies are in the ClinicalTrials.Gov database, they were registered after the results had been published. Following the implementation of ClinicalTrials.gov, investigators were required to prospectively declare their primary and secondary outcome variables. Prior to 2000, investigators had a greater opportunity to measure a range of variables and to select the most successful outcomes when reporting their results. For trials published before the year 2000, we found that 17 out of 30 (57%) reported significant benefit for their primary outcome. In the new era where primary outcomes are prospectively declared (published post 2000), only 2 of 25 trials (8%) reported a significant benefit (χ2 = 12.2, p = 0.0005).

Prospective declaration of the primary outcome variable is important because it eliminates the possibility of selecting for reporting an outcome among many different measures included in the study. In order to investigate this issue, we looked at the statistical significance of other variables not declared as the primary outcomes for preregistered studies. Among the 25 preregistered trials published in 2000 or later, 12 reported significant, positive effects for cardiovascular-related variables other than the primary outcome. Importantly, almost half of the trials might have been able to report a positive result if they had not declared a primary outcome in advance. Had the prospective declaration of a primary outcome not have been required, it is possible that the number of positive studies post-2000 would have looked very similar to the pre-2000 period.

Beginning in approximately 2000, the likelihood of showing a significant benefit in large NHLBI funded studies declined. Among the explanations we evaluated, the requirement of prospective registration in Clinicaltrials.gov is most strongly associated with the observed trend toward null clinical trials. The decline is not easily explained by the increased use of active comparators or a decline in industry sponsorship. In addition to the explanations at we evaluated using reported characteristics of the trials, we considered several other suggestions.

One explanation is that newer clinical trial management methodologies remove error variance and provide more precise estimates of treatment effects. If this were the explanation, refined methodologies and greater precision should have resulted in reductions in error variance, ultimately increasing the likelihood of finding treatments effects. But the probability of finding a treatment benefit decreased rather than increased as studies became more precise. As shown in Fig 1, variability in trial results declined systematically around the year 2000. As a result, we do not find better trial management to be a compelling explanation for the trend toward null results.

It is widely noted that journals favor publication of statistically significant findings[2]. Bias in favor of publishing positive outcomes is not a likely explanation for our results. We focused on large trials because previous analyses by NHLBI reported that 97% of trials with annual budgets over $500,000/year were published [3], thus removing publication bias as a rival explanation. In our analysis, 88% of the trials were published, although there may be a slight delay in the date of publication for null trials⁶. If positive trials are more likely to be published than null trials, we would have expected more positive published reports following 2000. A “file drawer” problem of suppressing null trial findings would result in over reporting positive results. Our observation of a trend toward null results goes in the opposite direction. If there is a bias, it is possible that stricter reporting standards and greater rigor in reporting requirements are suppressing the declaration of positive outcomes.

It has been argued that there have been few efficacious drugs in the pipeline[4,5]. Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6]. Thus, we would expect more null trials because the rate of developing effective new principals has declined. We believe this explanation unlikely because nearly all of the trials evaluated treatments that had been previously studied. For example, all of the treatments had been approved by the US FDA and these approvals require early phase trial evidence of safety and efficacy.

Another explanation for the increase in null trials is the possibility that medical care and supportive therapy have improved since 2000. As a result it has become difficult to demonstrate treatment effects because new approaches must compete with higher quality medical care. In support of this argument is the observation that outcomes in cardiovascular diseases continue to improve despite wide variation in the specific care that patients receive. On the other hand, outcomes of studies that compared treatment to an active standard of care comparison group achieved results quite similar to studies that compared treatment to placebo. However, we do recognize that the quality of background cardiovascular care continues to improve, making it increasingly difficult to demonstrate the incremental value of new treatments. The improvement in usual cardiovascular care could serve as alternative explanation for the trend toward null results in recent years.

Our results may also reflect greater involvement by NHLBI in trial design and execution. Prior to 2000, most large NHLBI clinical trials were investigator initiatLaed while nearly 80% of the trials published after 2000 had direct involvement of NHLBI through cooperative agreements. We recognize that industry sponsored trials may have a higher success rate. It is possible that industry conducts trials designed to demonstrate effectiveness while NHLBI uses its resources when there is true equipoise.

All post 2000 trials reported total mortality while total mortality was only reported in about 80% of the pre-2000 trials and many of the early trials were not powered to detect changes in mortality. The effects on total mortality were null for both pooled analyses of trials that were registered or not registered prior to publication (see data in online supplement) In addition, prior to 2000 and the implementation of Clinicaltrials.gov, investigators had the opportunity to change the p level or the directionality of their hypothesis post hoc. Further, they could create composite variables by adding variables together in a way that favored their hypothesis. Preregistration in ClinicalTrials.gov essentially eliminated this possibility.