Randomized controlled trials (trials) are often used to inform policy and practice for broad populations. A well designed and implemented trial allows consistent estimation of the average treatment/intervention effect (ATE) for the trial sample. We refer to this as the Study-specific ATE (SATE). If the question is whether that treatment should be used for people in a certain population (target population), then of interest is the Target population ATE (TATE). Since trial samples are often not representative of target populations, SATE may not be a good estimate of TATE. SATE departs from TATE if the trial sample differs from the target population with respect to the distribution of treatment effect modifiers.

Methods exist to use trial data to estimate TATE, e.g., [1–3], assuming treatment effect variation is explained by pre-treatment variables that are observed in both the trial and target population. Often, however, the variables observed for the target population are limited compared to those measured in the trial [4, 5]. This paper presents simple methods to assess the sensitivity of TATE estimates to effect modifiers observed in the trial but not in the target population. The paper starts from the simple case of additive effects based on an uncomplicated linear model (previously addressed in [6]) and extends to cases with more complex models and multiplicative effects. We clarify the assumptions of these methods for a general audience and provide detailed implementation instructions.

We illustrate the sensitivity analyses using an example based on the AIDS Clinical Trial Group (ACTG) 320 Study [7]. This trial randomized HIV-infected adults to two antiretroviral regimens: (i) two nucleoside reverse-transcriptase inhibitors (AZT or d4T and 3TC) plus a protease inhibitor (Indinavir), and (ii) AZT/d4T and 3TC only—referred to as new and old treatment, respectively. The trial found that relative to the old treatment, the new treatment lowered the hazard of AIDS and/or death. Cole & Stuart [1] generalized this effect to the population of people diagnosed with HIV in the US in 2006, using a set of covariates observed in both the trial and target population. Our example is based on this trial-population pair. We consider a different outcome, CD4 count (number of T-CD4 cells per ml blood), which is an important indicator of immunity status. We use the same target population dataset that Cole & Stuart created for [1] based on the CDC-estimated joint distribution of demographic characteristics in this population [8]. As the trial data are not for open access, for this illustration, we use a synthetic trial dataset created to mimic the distributions in the real trial data.

For this case, the basic mathematical results of the methods and one of their key assumptions were developed in [6]. The current paper elaborates on the full set of assumptions required by these methods, and pays close attention to details relevant to their effective use, such as different data scenarios and corresponding implementation instructions, including variation in weighting procedures for the method that involves weighting.

Turning our attention back to effect modifiers that are observed in the trial but not in the target population (V), note that the last section (and [6]) addressed a simple setting. We now offer two extensions of the sensitivity analyses to more complex situations.

The analyses presented here are merely illustrative. Results should not be taken as clinically informative. In addition to the concise presentation here, the detailed analyses can be found in S6 Appendix, with most of the code (R-code) included in the same appendix, and some Stata code in S7 Appendix. The data are provided in S8 and S9 Appendices.

The trial data include baseline and post-treatment CD4 counts and several baseline characteristics—age in years, sex, race, and severe immune suppression (SIS). The target population data include age groups, sex, and race. See Table 4 for a description of these variables. As noted above, when data include pre- and post-treatment measures of the outcome, there are more than one analysis options. Here we use random intercepts models on the combination of both measures. (For an application modeling post-treatment outcome only, see [6]). The definition of TATE is the average effect of treatment on potential CD4 count gain, and equivalently, the average effect of treatment on potential CD4 count post-treatment, in the target population.

We start by analyzing the trial data. The first question is whether to model CD4 count (and thereby consider its change) on the additive or multiplicative scale. CD4 count is a non-negative variable, so the additive scale may predict out of range, but it is also generally bounded above, which is more restrictive on multiplicative than on additive effects. We follow the HIV literature convention of using the additive scale for CD4 count, noting that our models may be suboptimal in this respect.

The two treatment arms in the trial are similar but the new treatment arm has more female patients and the old treatment arm has a higher SIS proportion (see Table 5). For the moment, assume we have good enough balance; we will come back to this.

SATE (i.e., the average difference, between the two treatments in the trial, in CD4 count change, or equivalently, in post-treatment CD4 counts) is estimated to be 36.6, 95% CI = (28.0,45.2) cells/ml by a simple model with no covariates, and 35.8, 95% CI = (27.3,44.4) by a model that adjusts for baseline covariates. Table 6 shows substantial differences in covariate distribution between the trial sample and target population, suggesting these SATE estimates should not be used as estimates of TATE.

Through a simple analysis examining interactions of FA with baseline covariates (e.g., FA*sex), covariate pairs (e.g., FA*sex*age), and cross-classifications of categorical covariates (e.g., FA*‘nonwhite-female’), we identify that the cross-classification of race and SIS status is an effect modifier (see Table 7). This variation in treatment effects is visualized in Fig 2.

As treatment effects vary across race-by-SIS-status categories, to obtain TATE estimates, we need the proportions of the target population that are in these categories. Race is observed in the target population, with 63.9% being nonwhite. SIS status (in the white and nonwhite groups), however, is not observed. If this were a substantive study, we would comb the literature to seek a plausible range for the proportions with SIS among White and among nonWhite people in the target population. As this is only illustrative, we specify a wide range (0.2, 0.6) for this proportion (which covers the proportion 0.456 in the trial), and assume that this proportion is the same between White and nonWhite people in the target population. With this assumption, this single proportion with SIS in the target population is now our sensitivity parameter.

A comment is warranted on assumption A3, that the range of the treatment effect modifiers in the target population is covered by the trial. In the current example, it is apparent that we do not have a problem with this assumption, because effect modification involves the four race-by-SIS-status categories, all of which are present in the trial sample. What might go unnoticed is that the age range in the trial sample (16 to 75) is smaller than that in the target population (13 to 80), and while we did not detect any treatment effect modification by age in the trial sample, we do not know if treatment effects for those younger than 16 or older than 75 (who are not represented in the trial sample) are the same as or different from treatment effects for those in the trial sample age range; this would be an area to consult clinical experts for their judgment. In the absence of such expert knowledge, were actual age values (rather than just age group) observed in the target population, one option would be to generalize treatment effects only to those aged 16 to 75 in the target population. This target population trimming is not an option in the current case as age values are not observed. In order to move forward, we assume that there is no effect modification by age.

We now apply the outcome-model-based sensitivity analysis (method 1) with the specified (0.2, 0.6) range for the sensitivity parameter (proportion with SIS in the target population). This obtains a range of TATE estimates from 36.2, 95% CI = (25.9,46.6) (corresponding to the lower end of the sensitivity parameter range) to 39.3, 95% CI = (30.0,48.6) (corresponding to the upper end of the sensitivity parameter range).

To apply the weighted-outcome-model-based sensitivity analysis (method 2), we need to weight the trial data to mimic the target population with respect to covariates observed in both (age group, sex, race). Heeding the recommendation of the two-step procedure, we first adjust the between-arms balance in the trial sample by inverse treatment propensity score weighting. Note that this results in a balance-adjusted SATE estimate of 33.9, 95% CI = (25.4, 42.3), which is very similar to (but slightly smaller than) the original SATE estimates. The effect modification model fit to the balance-adjusted trial data is very similar to that fit to the raw trial data. If we use that model and apply method 1, we obtain TATE estimates ranging from 34.2, 95% CI = (23.5, 44.8) to 37.1, 95% CI = (28.5, 45.7); these are slightly lower than those based on the model fit to the raw data. To use method 2, after adjusting within-trial balance, we weight the adjusted trial sample to the target population, using the weighting-by-the-odds procedure described in Table 3. This results in covariate balance shown in Table 8. The ATE in this weighted trial sample (which we will refer to as the (X, Z)- adjusted ATE) is 37.4, 95% CI = (27.0, 47.8), which is slightly higher than our SATE estimates. This is consistent with the fact that the weighted trial sample has higher nonWhite and SIS proportions than the original trial sample.

We now fit the same effect modification model to the weighted trial sample (see model fit in Table 9). Interestingly, the coefficients have changed substantially from the model fit to raw data (Table 7). This is not surprising as it could be a result of model misspecification (which we suspected), and it underscores our recommendation to use the weighted method so that the model that is the basis for estimating (sensitivity analysis of) TATE is the model fit to data that is close to the target population with respect to observed variables.

Generally we should not read much into the (lack of) statistical significance of interaction terms in the weighted model because weighting increases variance. However, the underwhelming coefficients in the fitted model above suggest that in the weighted dataset, there is not much differentiation of the group specific effects, and one could argue for using the (X, Z)-adjusted ATE as an estimate of TATE.

Or we could choose to proceed with a full implementation of method 2. Based on this effect modification model, we obtain TATE estimates ranging from 38.4, 95% CI = (26.0,50.9) (corresponding to the lower end, 0.20, of the range of the sensitivity parameter) to 37.5, 95% CI = (27.4,47.7) (corresponding to the upper end, 0.60, of the range of the sensitivity parameter).

Results from both methods are plotted in Fig 3. In this particular case, as there is not a clear slope of TATE estimates on the sensitivity parameter, we could combine these results to conclude that when between 20 and 60% of the target population have SIS, we could expect the target population average effect of the new treatment (relative to the old treatment) to be a gain in CD4 count of about 36 to 39 cells per ml (point estimate) with lower and upper confidence bounds of 26 and 51.

Policy-makers may want to use trial results to inform decisions for specific target populations. There are now methods to calibrate trial results to target populations, increasing potential for evidence-based decision-making [19]. However, these methods generally rely on a strong assumption that all effect modifiers differentially distributed between sample and population are observed and can be adjusted for. In this paper, we describe sensitivity analyses when this is not the case. Sensitivity analyses such as these allow researchers and decision-makers to have more appropriate confidence about population effects.

The methods proposed in this paper address situations where the effect definition matches the model assumed—additive effect with linear model, ratio effect with logit/log link model. While this kind of pairing is natural and commonplace, it is somewhat restrictive; there are situations where the effect definition of policy interest may not be on the same effect scale that best matches the model scientifically deemed appropriate. Our ongoing work aims to provide sensitivity analyses to situations where the model assumed is nonlinear but the effect scale of interest is additive.

That these sensitivity analyses rely on the assumption of a specific causal model deserves discussion, as the assumed model may or may not be close to the truth. Method 2, which combines this model with weighting, provides some protection against model misspecification, and thus has a flavor of double robustness. It is not truly doubly robust, however, because the weighting does not adjust for difference in the distribution of the partially unobserved effect modifier V. An area for further development is the search for sensitivity analysis procedures that are more flexible regarding model assumptions.

A challenge in generalization which we commented on is that the effect modifier range coverage assumption (A3) is perhaps commonly not met, because trial samples tend to be less diverse than target populations [20]. Strategies for this situation include combining evidence from multiple trials, combining experimental and non-experimental evidence using cross-design synthesis [21], or redefining the population to the area with overlap [22]. These complex approaches will require adapting sensitivity analyses.

Finally, it is important to note that these sensitivity analyses are not a panacea. It is best to limit the need for them in the first place. This can be done through careful consideration of target populations when designing trials, and efforts to enroll more representative trial samples. If that is not possible, trialists should carefully consider potential effect modifiers and investigate treatment effect heterogeneity. Also, to be able to adjust for effect modifiers when generalizing treatment effects, effect modifiers need to be measured consistently in trials and target population datasets [5, 23]. We encourage trialists to collect covariates in the same way as is done in common population datasets in their fields. When design strategies fall short, the methods discussed here provide a sense for the robustness (or lack thereof) of results when generalizing to target populations.