^{1}

^{1}

^{2}

^{3}

^{*}

^{4}

Conceived and designed the experiments: DB AWL JFW. Performed the experiments: DB JFW. Analyzed the data: DB AWL JFW. Contributed reagents/materials/analysis tools: DB AWL JFW. Wrote the paper: DB AWL JFW. Jointly posed the problem: AWL JFW. Provided the portfolio optimization framework: AWL DB. Provided medical expertise: JFW. Provided overall coordination for the project: AWL. Collected the raw NIH and YLL data: JFW. Jointly cleaned and processed the data: JFW DB. Reviewed the data: AWL JFW DB. Performed all computations under the supervision of AWL and JFW: DB. Discussed the results and implications and contributed original ideas to the project: AWL JFW DB. Prepared the final draft of the manuscript with contributions and editorial feedback from DB and JFW: AWL.

The authors have read the journal’s policy and have the following conflicts: the Massachusetts Institute of Technology Laboratory for Financial Engineering receives research grants from Citigroup and Bank of America and while the funds weren’t specifically designated for this study, a portion of those funds were contributed to a general pool that was directed to support a variety of faculty and student research projects, of which this study was one. AWL has the following declarations: (1) chairman, chief investment strategist, AlphaSimplex Group LLC (an asset management firm); (2) consultant, United States Treasury (Office of Financial Research); (3) member, Financial Advisory Roundtable, Federal Reserve Bank of New York; (4) member, Economic Advisory Committee, FINRA; (5) member, Academic Advisory and Research Committee, Moody’s; (6) member, Board of Trustees, Beth Israel Deaconess Medical Center; (7) research associate, National Bureau of Economic Research; (8) Research grant from the National Science Foundation on systemic risk; (9) Patent pending on cryptographic methods for computing systemic risk; (10) Patent awarded (U.S. Patent No. 7,599,876 October 6, 2009) “Electronic Market-Maker”; (11) Patent awarded (U.S. Patent No. 7,562,042 July 14, 2009) “Data Processor for Implementing Forecasting Algorithms”. There are no further patents, products in development or marketed products to declare. This does not alter the authors’ adherence to all the PLoS ONE policies on sharing data and materials, as detailed online in the guide for authors.

The National Institutes of Health (NIH) is among the world’s largest investors in biomedical research, with a mandate to: “…lengthen life, and reduce the burdens of illness and disability.” Its funding decisions have been criticized as insufficiently focused on disease burden. We hypothesize that modern portfolio theory can create a closer link between basic research and outcome, and offer insight into basic-science related improvements in public health. We propose portfolio theory as a systematic framework for making biomedical funding allocation decisions–one that is directly tied to the risk/reward trade-off of burden-of-disease outcomes.

Using data from 1965 to 2007, we provide estimates of the NIH “efficient frontier”, the set of funding allocations across 7 groups of disease-oriented NIH institutes that yield the greatest expected return on investment for a given level of risk, where return on investment is measured by subsequent impact on U.S. years of life lost (YLL). The results suggest that NIH may be actively managing its research risk, given that the volatility of its current allocation is 17% less than that of an equal-allocation portfolio with similar expected returns. The estimated efficient frontier suggests that further improvements in expected return (89% to 119% vs. current) or reduction in risk (22% to 35% vs. current) are available holding risk or expected return, respectively, constant, and that 28% to 89% greater decrease in average years-of-life-lost per unit risk may be achievable. However, these results also reflect the imprecision of YLL as a measure of disease burden, the noisy statistical link between basic research and YLL, and other known limitations of portfolio theory itself.

Our analysis is intended to serve as a proof-of-concept and starting point for applying quantitative methods to allocating biomedical research funding that are objective, systematic, transparent, repeatable, and expressly designed to reduce the burden of disease. By approaching funding decisions in a more analytical fashion, it may be possible to improve their ultimate outcomes while reducing unintended consequences.

The National Institutes of Health (NIH) is among the world’s largest and most important investors in biomedical research. Its stated mission is to “seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce the burdens of illness and disability” (

We consider a framework in which biomedical research allocation decisions are more directly tied to the risk/reward trade-off of burden-of-disease outcomes. Prioritizing research efforts is analogous to managing an investment portfolio: in both cases, there are competing opportunities to invest limited resources, and expected returns, risk, correlations, and the cost of lost opportunities are important factors in determining the return of those investments.

Financial decisions are commonly made according to portfolio theory

We recast the NIH funding allocation decision as a portfolio-optimization problem in which the objective is to allocate a fixed amount of funds across a set of disease groups to maximize the expected “return on investment” (ROI) for a given level of volatility. We define ROI as the subsequent improvements in years of life lost (YLL), and using historical time series data provided by the NIH (

Portfolio theory highlights the value of diversification: investing in multiple securities with imperfectly correlated pay-offs almost always yields a better reward-to-risk profile than investing in a single security. For developing this framework, Markowitz shared the Nobel Memorial Prize in Economic Sciences, and today portfolio theory is the starting point for investment management decisions among the largest institutional investors

The NIH has 27 Institutes and Centers, of which we identified 10 with research missions clearly tied to specific disease states, and which account for $21 billion of funding in 2005 or 74% of the total. The disease classification scheme used and the procedure for constructing the appropriation time series are described in greater detail in

NIH appropriations in real (2005) dollars, categorized by disease group (

These Institutes and the basic research they fund have inevitable overlap and effect beyond their charter; we treat all spending for any given Institute as being directed toward the corresponding disease states, and account for spillover effects by considering the correlations in the lessening of the burden of disease in other groups. For example, molecular biology funded by the NCI may be relevant to infectious diseases but, like the entire NCI budget, would be assumed for modeling purposes to be directed at cancer; the hypothetical infectious-disease improvement would appear in the correlation between the decrease in years of life lost for cancer and that of infectious diseases.

Because of its simplicity, availability, breadth, and long history, years of life lost (YLL) was chosen as the measure of burden of disease to be used in constructing the estimated return on investment from NIH-funded research (see

Using 2005 as the base year, the raw YLL observations were adjusted in other years to be comparable to the 2005 population:

Panel (a): Raw YLL categorized by disease group (

Three disease areas required special consideration: HIV, AMS, and dementia. AMS and HIV have shorter histories, which is problematic for estimating parameters based on historical returns that are lagged by typical FDA approval times plus 4 years. Dementia, including Alzheimer’s disease and unspecified psychoses, was reclassified with the change from ICD-9 to ICD-10 from mental and behavioral disorders to diseases of the nervous system; we placed all dementia YLL in the CNS group to avoid a transition-point artifact at the juncture between ICD-9 and ICD-10, and then performed a sensitivity analysis with and without the dementia YLL. Further work could, in a manner analogous to our treatment of HIV, treat this group of neurodegenerative conditions as a separate category.

HIV poses a special challenge given its extreme returns after the introduction of protease inhibitors, which are outliers that are likely to be non-stationary and would heavily bias the parameter estimates on which the portfolio optimization is based. To address this outlier, HIV spending and its corresponding YLL were omitted from those of other infectious diseases–the component of NIAID spending directed at HIV was estimated by straight-line interpolation from published figures, and this HIV spending was treated as a separate entity and subtracted from reported NIAID appropriations; a similar procedure was followed for the estimation of HIV-related YLL, and WONDER was queried at the subchapter level to implement this separation.

For completeness, empirical results that include AMS and HIV data are provided in

To apply portfolio theory, the concept of a “return on investment” (ROI) must first be defined. Although YLL has already been chosen as the metric by which the impact of research funding is to be gauged, there are at least two issues in determining the relation between research expenditures and YLL that must be considered. The first is whether or not any relation exists between the two quantities. While the objectives of pure science do not always include practical applications that impact YLL, the fact that part of the NIH mission is to “reduce the burdens of illness and disability” suggests the presumption–at least by the NIH–that there is indeed a non-trivial relation between NIH-funded research and burden of disease. For the purposes of this study, and as a first approximation, we assume that YLL improvements are proportional to research expenditures. Of course, factors other than NIH research expenditures also affect YLL, including research from other domestic and international medical centers and institutes, spending in the pharmaceutical and biotechnology industries, public health policy, behavioral patterns, prosperity level and environmental conditions. Therefore, the YLL/NIH-funding relation is likely to be noisy, with confounding effects that may not be easily disentangled. See the

The second issue is the significant time lag between research expenditures and observable impact on YLL. For example, Mosteller

The impact of NIH-funded research on disease burden is likely to be spread out over several years after this intervening lag, given the diffusion-like process in which research results are shared in the scientific community. For simplicity, the same duration (^{2} and the corresponding lags are shown in

Summary statistics for the ROI of disease groups, in units of years (for the lag length) and per-capita-GDP-denominated reductions in YLL between years

This procedure is, of course, a crude but systematic heuristic for relating research funding to YLL outcomes. Alternatives include using a single fixed lag across all groups, simply assuming particular values for group-specific lags based on NIH mandates and experience, computing a time-weighted average YLL for each group with a weighting scheme corresponding to an assumed or estimated knowledge-diffusion rate for that group, or constructing a more accurate YLL return series by tracking individual NIH grants within each group to determine the specific impact on YLL (through new drugs, protocols, and other improvements in morbidity and mortality) from the award dates to the present. While the choice of lag is critical in determining the characteristics of the YLL return series and deserves further research, it does not effect the applicability of the overall analytical framework. While our procedure is surely imperfect, it is a plausible starting point from which improvements can be made.

Assuming constant impact of research funding on YLL over the duration of

where the minus sign reflects the focus on

Given the definition in equation (2) for the ROI of each of the disease groups, the “optimal” appropriation of funds among those groups must be determined, i.e., the appropriation that produces the best possible aggregate expected return on total research funding per unit risk. Denote by _{p}

In some financial applications, a variation of this optimization problem is employed in which the objective function is augmented to include penalty for allocations that deviate from some pre-specified vector of target weights

Summary statistics of the ROI for the period 1980–2003 are presented in

Negative mean ROIs are counterintuitive–implying that increasing investment is counterproductive to easing the burden of disease–yet they occur in three disease groups: AID, CNS, and DDK. There are several reasons for this phenomenon. First, and foremost, unlike investments in financial assets, there is significant randomness in the relation between NIH spending and subsequent impact on YLL. Many factors other than the amount of funding affect the success or failure of pure and translational research, and average ROI values reflect the impact of all of those factors. For example, in the case of CNS–in which the negative return is more than two standard deviations away from 0–there is a large non-stationary effect due to the rapid growth of a group of dementias. A sensitivity analysis confirms that the negative returns are largely due to the dementia effect as is indicated in sub-panels (c) and (d) of

Efficient frontiers for (a) all groups except HIV and AMS,

A more subtle effect comes from the fact that favorable ROIs in one area can impart negative bias in other groups, since all deaths must be assigned to one cause or group. Consider a simple thought experiment in which only two lethal diseases, A and B, exist. If a cure for A is discovered, then those who would otherwise have died of A must necessarily die of B eventually. This yields an increase in the YLL for B, even if the treatment of B diseases has not worsened. Similar, if less-extreme, dynamics can emerge with more disease groups and less-dramatic progress that merely reduces rather than eliminates the YLL burden of a specific disease group.

Finally, and perhaps least likely, the dissemination of erroneous research results

Rather than “correcting” these counterintuitive empirical relations, we view them as important anomalies that deserve further scrutiny and analysis. In some cases, e.g., CNS, the anomaly can be traced to a specific external factor that can either be accepted as legitimate or set aside as an extreme outlier that is not representative of the true relation between funding and subsequent YLL. In the latter case, one alternative to using an empirically estimated mean ROI to compute the efficient frontier is to impose a Bayesian prior on this parameter (see

To develop intuition for possible patterns between funding allocation and improvements in YLL, the cumulative sums of these two variables are plotted in

In

historical average NIH allocation for years 1996–2005;

equal-weighted (

minimum-variance allocation;

the allocation on the efficient frontier that has the same mean as the average NIH allocation (the “NIH-mean” allocation);

the allocation on the efficient frontier which has the same variance as the average NIH allocation (the “NIH-var” allocation);

the allocation on the efficient frontier that is 25% of the distance from the minimum variance allocation to the maximum expected-return allocation;

the allocation on the efficient frontier that is 50% of the distance from the minimum variance allocation to the maximum expected-return allocation;

the allocation on the efficient frontier that is 75% of the distance from the minimum variance allocation to the maximum expected-return allocation.

The region bounded by the horizontal segment (

Benchmark, single- and dual-objective optimal portfolio weights (in percent), based on historical ROI from 1980 to 2003.

Relative volatility (σ), expected return (µ) and risk-adjusted returns (µ/σ) for different scenarios (see text) for both

A sensitivity analysis is conducted by estimating the efficient frontier with (

The top left sub-panel of

However, the dementia effect may underestimate the performance of the CNS disease group, hence the lower panel of

Portfolio theory is a systematic framework for determining optimal research funding allocations based on historical return on investment, variance, and correlation between appropriations and reductions in disease burden. The optimization results suggest that significant YLL improvements with respect to a mean-variance criterion may be possible through funding re-allocation. To our knowledge, this is the first time such an approach has been empirically implemented in this domain.

Our method identifies optimal portfolios which, for a given degree of risk, are efficient in reducing YLL. However, some optimal allocations may allocate no funds to certain disease groups (typically those with low expected return and high volatility). While this may be reasonable from a mean-variance optimization perspective, it is obviously an extreme and impractical outcome. A first step toward recognizing the trade-off between reallocation costs and efficiency is the dual-objective optimization procedure which penalizes allocations away from a pre-specified “benchmark” allocation, e.g., the current NIH policy. This more-conservative approach increases diversification with less reallocation cost than the single-objective procedure. For instance, in the dual-objective optimization, the NIH-mean portfolio is relatively close to the current allocation (see

Our findings must be qualified in at least three respects: (1) YLL as a measure of burden of disease, which is clearly incomplete and less than ideal; (2) the definition of ROI and the challenges of relating research expenditures to subsequent outcomes such as burden of disease; and (3) the known limitations of portfolio theory from the financial context. Each of these qualifications is discussed in greater detail in

YLL captures only the most extreme form of disease burden; more refined measures such as disability-adjusted or quality-adjusted life years are clearly preferable. However, time series histories for such measures are currently unavailable. Therefore, YLL is the most natural starting point for gauging the impact of biomedical research funding, and is directly aligned with the NIH mission to “lengthen life”.

Our definition of ROI can also be challenged as being imprecise and

While all of these qualifications have merit, they are not insurmountable obstacles and can likely be addressed through additional data collection and more sophisticated metrics, perhaps along the lines of Porter

There are also several limitations of portfolio theory that are well-known in the financial context (e.g., estimation error, parameter instability, and exogenous constraints such as non-negativity restrictions on portfolio weight), all of which can be addressed to some degree through statistical techniques such as resampling, Bayesian analysis, and robust optimization

This critical step is a pre-requisite to any formal analysis of funding allocation decisions, and underscores the need for integration of basic science with biomedical investment performance analysis and science policy. Such integration will require close and ongoing collaboration between scientists and policymakers to determine the appropriate parameters for the funding allocation process, and to incorporate prior information and qualitative judgments

These qualifications suggest that portfolio theory cannot be mechanically applied to historical data to yield actionable optimal allocations. However, our empirical results should be sufficient proof-of-concept to motivate additional data collection, empirical analysis, and research to advance the state of the art in this application area. Any repeatable and transparent process for making funding allocation decisions–especially one that involves criteria other than peer-review-based academic excellence–will, understandably, be viewed initially with some degree of suspicion and contempt by the scientific community. But if one of the goals of biomedical research is to reduce the burden of disease, some tension between academics and public policy may be unavoidable. Moreover, in the absence of a common framework for evaluating the trade-offs between academic excellence and therapeutic potential, other proposed alternatives such as political earmarking

In an environment of tightening budgets and increasing oversight of appropriations, portfolio theory offers scientists, policymakers, and regulators–all of whom are, in effect, research portfolio managers–a rational, systematic, transparent, and reproducible framework in which to explicitly balance expected benefits against potential risks while accounting for correlation among multiple research agendas and real-world constraints in allocating scarce resources. Most funding agencies and scientists have already been making such trade-offs informally and heuristically. There may be additional benefits to making such decisions within an explicit framework based on standardized and objective metrics.

One of the most significant benefits from adopting such a framework may be the reduction of uncertainty surrounding future funding-allocation decisions. This alone would greatly enhance the ability of funding agencies and scientists to plan for the future and better manage their respective budgets, research agendas, and careers. By approaching funding decisions in a more analytical fashion, it may be possible to improve their ultimate outcomes while reducing the chances of unintended consequences.

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(EPS)

(PDF)

We thank Melissa Antman, Ernie Berndt, Stefano Bertuzzi, Martin Brown, Carole Christian, Charles DeLisi, Anne Fladger, Huybert Groenendaal, Sushil Gupta, Robert Harriman, Lynn Hudson, Srinivasan Kumar, Eric Lander, Meaghan Muir, Chris Murray, Luci Roberts, Carl Roth, James Schuttinga, David Soybel, Narayan Venkatasubramanyan, and Madeleine Wallace for comments and discussion. The views and opinions expressed in this article are those of the authors only, and do not necessarily represent the views and opinions of: the individuals acknowledged above, AlphaSimplex Group, Brigham and Women’s Hospital, Harvard Medical School, MIT, or any of their affiliates and employees.