PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA PONE-D-24-04044 10.1371/journal.pone.0308543 Study Protocol Research and analysis methodsSimulation and modeling Research and analysis methodsMathematical and statistical techniquesStatistical methodsRegression analysisLinear regression analysis Physical sciencesMathematicsStatisticsStatistical methodsRegression analysisLinear regression analysis Research and analysis methodsMathematical and statistical techniquesStatistical methodsMonte Carlo method Physical sciencesMathematicsStatisticsStatistical methodsMonte Carlo method Research and analysis methodsMathematical and statistical techniquesStatistical methodsForecasting Physical sciencesMathematicsStatisticsStatistical methodsForecasting Research and analysis methodsResearch assessmentReproducibility Research and analysis methodsResearch assessmentResearch errors Research and analysis methodsMathematical and statistical techniquesStatistical methodsRegression analysis Physical sciencesMathematicsStatisticsStatistical methodsRegression analysis Biology and life sciencesNutrition Medicine and health sciencesNutrition Evaluating variable selection methods for multivariable regression models: A simulation study protocol Evaluating variable selection methods for multivariable regression models https://orcid.org/0000-0003-1215-8561 Ullmann Theresa Conceptualization Methodology Software Visualization Writing – original draft Writing – review & editing 1 Heinze Georg Conceptualization Funding acquisition Methodology Supervision Writing – original draft Writing – review & editing 1 Hafermann Lorena Writing – review & editing 2 https://orcid.org/0000-0003-3943-6234 Schilhart-Wallisch Christine Software Writing – review & editing 1 3 https://orcid.org/0000-0003-1339-0311 Dunkler Daniela Conceptualization Funding acquisition Methodology Supervision Writing – original draft Writing – review & editing 1 * for TG2 of the STRATOS initiative Institute of Clinical Biometrics, Center for Medical Data Science, Medical University of Vienna, Vienna, Austria Institute of Biometry and Clinical Epidemiology, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany Austrian Agency for Health and Food Safety (AGES), Vienna, Austria Tian Suyan Editor The First Hospital of Jilin University, CHINA

The authors have declared that no competing interests exist.

¶ Membership list can be found in the Acknowledgments section.

 * E-mail: daniela.dunkler@meduniwien.ac.at 2024 9 8 2024 19 8 e0308543 7 2 2024 25 7 2024 2024 Ullmann et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Researchers often perform data-driven variable selection when modeling the associations between an outcome and multiple independent variables in regression analysis. Variable selection may improve the interpretability, parsimony and/or predictive accuracy of a model. Yet variable selection can also have negative consequences, such as false exclusion of important variables or inclusion of noise variables, biased estimation of regression coefficients, underestimated standard errors and invalid confidence intervals, as well as model instability. While the potential advantages and disadvantages of variable selection have been discussed in the literature for decades, few large-scale simulation studies have neutrally compared data-driven variable selection methods with respect to their consequences for the resulting models. We present the protocol for a simulation study that will evaluate different variable selection methods: forward selection, stepwise forward selection, backward elimination, augmented backward elimination, univariable selection, univariable selection followed by backward elimination, and penalized likelihood approaches (Lasso, relaxed Lasso, adaptive Lasso). These methods will be compared with respect to false inclusion and/or exclusion of variables, consequences on bias and variance of the estimated regression coefficients, the validity of the confidence intervals for the coefficients, the accuracy of the estimated variable importance ranking, and the predictive performance of the selected models. We consider both linear and logistic regression in a low-dimensional setting (20 independent variables with 10 true predictors and 10 noise variables). The simulation will be based on real-world data from the National Health and Nutrition Examination Survey (NHANES). Publishing this study protocol ahead of performing the simulation increases transparency and allows integrating the perspective of other experts into the study design.

 http://dx.doi.org/10.13039/501100002428 Austrian Science Fund I-4739-B https://orcid.org/0000-0003-1339-0311 Dunkler Daniela http://dx.doi.org/10.13039/501100001659 Deutsche Forschungsgemeinschaft RA 2347/8-1 This research was funded in part by the Austrian Science Fund (FWF, https://www.fwf.ac.at/en/) [I-4739-B] (for T.U. and C.W.) and by the German Research Foundation (DFG, https://www.dfg.de/en) [RA 2347/8-1] (for L. H.). For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. The funders did not and will not have any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability This manuscript is a protocol of a simulation study. We intend to share the software code after the study has been conducted and published. This will allow recreating our data and reproducing our simulation study. 1 Introduction

Data-driven variable selection is frequently performed when modeling the associations between an outcome and multiple independent variables (sometimes also referred to as explanatory variables, covariates or predictors). Variable selection may help to generate parsimonious and interpretable models, and may also yield models with increased predictive accuracy. Despite these potential advantages, data-driven variable selection can also have unintended negative consequences that many researchers are not fully aware of. Variable selection induces additional uncertainty in the estimation process and may cause biased estimation of regression coefficients, model instability (i.e., models that are not robust with respect to small perturbations of the data set), and issues with post-selection inference such as underestimated standard errors and invalid confidence intervals [1–5].

A recent review [1] provided guidance about variable selection and gave an overview of possible consequences of variable selection. However, there are few systematic simulation studies that compare different variable selection methods with respect to their consequences for the resulting models (for some exceptions, see [6–10]). While many articles proposing new variable selection methods include a comparison with existing methods (based on simulated or real data), these comparisons are typically somewhat limited, often comparing the new method to only one to three competitors, even though there are many more existing methods. Moreover, these articles are inherently biased towards demonstrating superiority of the new methods. In particular, such studies cannot be considered as neutral. A neutral comparison study is a study whose authors do not have a vested interest in one of the competing methods, and are (as a group) approximately equally familiar with all considered methods [11, 12]. More neutral comparison studies about existing variable selection methods are needed to better understand their properties, a viewpoint that aligns with the goals of the STRATOS initiative (STRengthening Analytical Thinking for Observational Studies [13]). The STRATOS initiative is an international consortium of biostatistical experts, and aims to provide guidance in the design and analysis of observational studies for specialist and non-specialist audiences. This perspective motivates our comprehensive simulation study.

We will focus on descriptive modeling (i.e., describing the relationship between the outcome and the independent variables in a parsimonious manner) and predictive modeling (i.e., predicting the outcome as accurately as possible) [14]. Our setting is multivariable regression analysis with one outcome variable. The outcome is either continuous (linear regression) or binary (logistic regression). We simulate data in a low-dimensional scenario (20 variables consisting of 10 true predictors and 10 noise variables). Different variable selection methods with multiple parameter settings are compared: forward selection, stepwise forward selection, backward elimination, augmented backward elimination [15], univariable selection, univariable selection followed by backward elimination, the Lasso [16], the relaxed Lasso [9, 17], and the adaptive Lasso [18]. We compare the performances of these methods with respect to false inclusion and/or exclusion of variables, consequences on bias and variance of the estimated regression coefficients, the validity of the confidence intervals for the coefficients, the accuracy of the estimated variable importance ranking, and finally the predictive performance of the selected models.

Using simulated instead of real data allows us to a) know the true data generating process and b) systematically vary several data characteristics [19, 20]. For example, we will include varying sample sizes and R2, as the consequences of variable selection depend on these parameters. To ensure that the simulation results are practically relevant, we use real data as the starting point for our simulation. The distributions and correlation structure of the variables are based on data from the National Health and Nutrition Examination Survey (NHANES) [21]. The choice of variables and true regression coefficients is inspired by an applied study about predicting the difference between ambulatory/home and clinic blood pressure readings [22]. Our simulated data thus mimics real cardiovascular data.

Our focus is on low-dimensional data, which is reflected in our simulation setting with twenty independent variables. Data of this type frequently appears in medicine and other application fields, and researchers often apply variable selection in this context. For example, a systematic review of models for COVID-19 prognosis [23, 24] identified 236 newly developed regression models for prediction. Data-driven variable selection was applied (and reported) for 196 models. In 165 models both the number of candidate predictors (i.e., the predictors considered at the start of data-driven selection) and the number of predictors in the final model were reported; the median numbers were 28 (range 4–130), and 6 (range 1–38), respectively. This demonstrates that low- to medium-dimensional data played an important role in COVID-19 prediction research. Of course, data-driven variable selection is also relevant for high-dimensional data. Comparing variable selection methods for high-dimensional data would require a different study design and is not the purpose of this planned simulation study.

As mentioned above, neutrality is an important goal when conducting systematic comparison studies. “Perfect” neutrality may be the ultimate goal, but this ideal can be difficult to achieve in practice. While we aim to be as neutral as possible, we disclose (for the purpose of full transparency) that one of the methods for variable selection included in our comparison, namely augmented backwards elimination, was originally proposed by two authors of the present study protocol [15]. Our goal was to not let this fact influence our choice of study design, though unconscious biases can never be fully excluded. Striving for as much neutrality as possible motivated us to publish this study protocol. This will allow us to integrate the comments of reviewers before performing the simulation. For the design of our study, results from previous smaller simulation studies and pilot studies were taken into account [1]; however, the study outlined in this protocol has not yet been run and analyzed. Preregistration of study protocols for simulation studies/methodological studies is still very rare (for an exception, see [25]). However, this practice could offer similar advantages to those discussed for preregistration in applied research, such as increased transparency and prevention of “hindsight bias” [26]. Potential advantages of preregistering protocols for simulation studies, but also possible limitations and challenges, are discussed more extensively elsewhere [27].

A specific goal of our simulation study is to evaluate previously published recommendations about variable selection [1], which we discuss in Section 2. We then describe our simulation design in Section 3, explain the planned code review in Section 4, and conclude the protocol with some final remarks in Section 5.

 2 Previous variable selection recommendations

Varied viewpoints exist in the literature as to whether researchers should apply data-driven variable selection, and, if so, which methods and parameters are deemed preferable. Some authors generally caution against data-driven variable selection, stressing potential negative consequences [5]. Other authors put more focus on potential advantages of variable selection and are more optimistic about using selection methods, at least if the sample size is large enough and if selection is accompanied by a stability analysis [28]. In a review conducted by three co-authors of the present study protocol, Heinze et al. [1] summarized different perspectives from the literature. Drawing upon existing recommendations, but also taking their own experience and a small simulation study into account, they derived recommendations for the usage of variable selection methods. These recommendations consider both benefits and drawbacks of variable selection, thereby reconciling different viewpoints on the matter. The recommendations depend on the “events-per-variable” (EPV) in the data. The EPV is the ratio between sample size (in linear regression) or the number of the less frequent outcome (in logistic regression) and the number of independent variables. Data-driven variable selection is applied on a carefully designed “global” model which includes all independent variables relevant for the research question. The denominator of EPV refers to the number of design variables (including possible dummy variables and other constructed variables) in this global model. The following bullet points list the recommendations, and how we plan to evaluate them.

EPV > 25: While variable selection may generally work well for a large EPV value, the selection of independent variables with small effect size can still be unstable. If backward elimination is used, a stringent threshold of α = 0.05 or selection with the BIC may lead to a more accurate selection of variables than milder thresholds.

In our study: We will check whether selection rates of variables with small standardized regression coefficients (e.g., ±0.25) are notably different from either 0 or 1 (which indicates instability). For backward elimination, we will evaluate whether the selection of variables is more accurate when using the threshold α = 0.05 or the BIC (which corresponds to even stricter thresholds for our considered sample sizes [1]), compared to using larger α values.

10 < EPV ≤ 25: In general, the selection of variables might be unstable with such an EPV. When variables with unclear effect size are selected, their effects might be over-estimated. Penalized estimation (Lasso) or postestimation shrinkage is thus recommended. If backward elimination is used, a threshold corresponding to selection with the AIC (approximately α = 0.157) is recommended, but not smaller α values.

In our study: Again, we will evaluate stability by checking whether selection rates of variables, particularly those with small standardized regression coefficients, are notably different from either 0 or 1. We will also calculate the conditional bias (i.e., bias conditioned on selection) of the variables and analyze whether variables with small standardized regression coefficients have large conditional bias away from zero. For backward elimination, we will evaluate to which extent a threshold of α = 0.157 (or an even milder threshold of α = 0.5) selects the true predictors more frequently than smaller thresholds (i.e., a fixed threshold of α = 0.05 or selection with the BIC) [3].

EPV ≤ 10: Data-driven variable selection is generally not recommended.

In our study: We will analyze whether variable selection has negative consequences with respect to the different performance criteria.

The results of variable selection are not only influenced by EPV, but also by other aspects such as the R2 of the model. We will thus consider different R2 values in our simulation study. The recommendations above do not take R2 into account, as the R2 of the model is typically not known prior to the data analysis.

 3 Simulation design

Morris et al. [19] proposed to describe the following components when reporting a simulation study: the aims of the study (A), the data-generating mechanisms (D), the estimands (i.e., the population quantities which are estimated) and other targets of interest (E), the methods to be compared (M), and the performance measures used for evaluating the methods (P). The ADEMP components of our study are briefly summarized in Tables 1 and 2. We now describe the components in more detail.

10.1371/journal.pone.0308543.t001 Summary of the simulation design, part 1: Aims and data-generating mechanisms.
Aims (Section 3.1) Comparison of popular data-driven variable selection methods for multivariable linear or logistic regression, with respect to their consequences for the resulting models.
Data-generating mechanisms (Section 3.2)

20 variables: 10 predictors X1, …, X10 and 10 noise variables X11, …, X20 (mixture of binary and continuous variables)

Distributions and correlation structure of the variables are based on NHANES data [21] (see Fig 1, S1 Fig and S1 Table).

Standardized regression coefficients for the predictors X1, …, X10: ( β 1 s d , … , β 10 s d ) = ( 1 . 5 , - 1 , 1 , 0 . 75 , 0 . 5 , 0 . 5 , 0 . 5 , - 0 . 5 , - 0 . 25 , - 0 . 25 ) The regression coefficients for X11, …, X20 are set to zero.

For settings with linear effects, the outcome Y is simulated as follows:

For linear regression: Y = xβ + ϵ with ϵ ∼ N(0, σ2), and σ2 chosen such that R2 = 0.45 (setting 1), R2 = 0.15 (setting 2), or R2 = 0.7 (setting 3). The intercept β0 is set to 36.

For logistic regression: outcomes Y are drawn from a Bernoulli distribution with event probability 1/(1 + exp(−c xβ)). The intercept β0 and the constant c > 0 are adjusted such that

the expected event probability equals 0.3 with Cox-Snell R C S 2 = 0 . 40 (setting 4) or R C S 2 = 0 . 13 (setting 5)

the expected event probability equals 0.05 with Cox-Snell R C S 2 = 0 . 16 (setting 6) or R C S 2 = 0 . 05 (setting 7)

To evaluate models with mildly misspecified functional forms (a frequent situation in practice), each of the 7 settings is also considered with Y generated with nonlinear effects, yielding 14 settings in total (see Table 3).

Additionally, we consider three simplified scenarios with all variables N ( 0 , 1 )-distributed and/or uncorrelated (see Section 3.2.5 for details).

Each setting is considered with varying sample sizes (see Table 4).
10.1371/journal.pone.0308543.t002 Summary of the simulation design, part 2: Estimands, methods, and performance measures.
Estimands and other targets (Section 3.3) Regression coefficients, model selection, predictions
Methods (Section 3.4)

Forward selection with AIC

Stepwise forward selection with AIC (forward selection with backward elimination steps)

Backward elimination with α = 0.05, α = 0.5, AIC, or BIC

Augmented backward elimination (ABE) with AIC and τ = 0.05

Univariable selection with α = 0.05 or α = 0.20

Univariable selection with α = 0.20 followed by backward elimination with α = 0.05

Lasso with λ tuned with 10-fold cross-validation

Relaxed Lasso with λ tuned with 10-fold cross-validation or BIC

Adaptive Lasso with λ tuned with 10-fold cross-validation

 We also consider the global model with all variables.
Performance measures (Section 3.5) Mainly for descriptive models:

Bias and root of mean squared error of the coefficients

Coverage and width of the 95% confidence intervals for the coefficients

Type 1 error rate/power

False positive rate/true positive rate

Kendall’s τB for variable rankings

Selection rate of the true model, of an over-selection model, and of an under-selection model

 Mainly for prediction models:

Local bias and local root mean squared error w.r.t. estimated vs. true linear predictor

Root mean squared error and median absolute error of estimated vs. true linear predictor, additionally AUC for logistic regression

Integrated calibration index (ICI)
3.1 Aims (A)

We aim to compare different variable selection methods for multivariable linear or logistic regression, with respect to their consequences for the resulting models. We consider consequences on bias and variance of the estimated regression coefficients, validity of confidence intervals for the coefficients, false inclusion or exclusion of variables, and predictive performance. We analyze the behavior of variable selection methods…

… depending on sample size/EPV, with particular focus on evaluating the recommendations of Heinze et al. [1],

… depending on the R2 of the population model,

… depending on the modeling goal (description or prediction),

… when functional forms are misspecified (i.e., when fitting models assuming linear functional forms of continuous predictors even though the true functional forms are nonlinear),

… when switching from our realistic scenario that mimics cardiovascular data to simplified scenarios (i.e., all variables are normally distributed and/or uncorrelated).

 3.2 Data-generating mechanisms (D) 3.2.1 Simulation of independent variables (predictors and noise variables)

We simulate 20 independent variables: 10 true predictors (from now on just called “predictors”) and 10 noise variables. The correlation structure and distributions are based on real-world data from the 2013–14 and 2015–2016 cycles of the National Health and Nutrition Examination Survey (NHANES) [21]. To choose suitable variables in the NHANES data, we drew inspiration from a regression model reported by Sheppard et al. [22] for predicting the difference between diastolic blood pressure readings as measured ambulatory/at home versus in the clinic. The variables are described in detail in S1 Appendix. The correlation matrix Σ for the simulation is based on the empirical correlation matrix of the variables. For better interpretability, we set correlations below 0.15 to zero and round all values to the closest multiple of 0.05 (see S1 Fig and S1 Table for the resulting correlation matrix).

To obtain distributions from the NHANES data, we fit Bernoulli distributions for the binary variables, and normal distributions, log-normal distributions, or approximations of the empirical cumulative distribution function (CDF) for the continuous variables. For each continuous variable, we truncate its distribution with the minimum of the variable in the NHANES data as the lower bound and the maximum as the upper bound. The resulting distributions are as follows (see also Fig 1):

predictors: X1 (log-normal), X2 (continuous with approximated CDF), X3 (log-normal), X4 (binary, p = 0.50), X5 (normal), X6 (binary, p = 0.29), X7 (log-normal), X8 (log-normal), X9 (normal), X10 (binary, p = 0.11)

noise variables: X11 (log-normal), X12 (normal), X13 (log-normal), X14 (binary, p = 0.61), X15 (normal), X16 (binary, p = 0.20), X17 (log-normal), X18 (normal), X19 (normal), X20 (binary, p = 0.20)

 10.1371/journal.pone.0308543.g001 Distributions and pre-specified standardized regression coefficients of predictors and noise variables.

Predictors are ordered by absolute values of standardized regression coefficients. Histograms are based on a large simulated dataset (n = 100, 000).

The distributions, together with the correlation matrix Σ, are then used as input for the normal-to-anything (NORTA) method for simulation [29, 30].

 3.2.2 Choice of regression coefficients

For choosing the standardized regression coefficients of the predictors X1, …, X10, we drew inspiration from the coefficients reported for the regression model of Sheppard et al. [22]: ( β 1 s d , … , β 10 s d) t = ( 1 . 5 , - 1 , 1 , 0 . 75 , 0 . 5 , 0 . 5 , 0 . 5 , - 0 . 5 , - 0 . 25 , - 0 . 25 )

This choice reflects a mixture of stronger and weaker effects, a situation typical for many applications in biology and medicine. We would expect different behaviors of the predictors during variable selection depending on their effects.

The standardized coefficients β j s d are transformed into non-standardized coefficients βj as follows: standard deviations (SDs) of the variables are calculated based on a single large simulated dataset DP to approximate the population (n = 100, 000) and the standardized coefficients are divided by these SDs.

The regression coefficients for the noise variables X11, …, X20 are set to zero.

As intended, there is no systematic association between the absolute values | β j s d | and the coefficients of determination R j 2 for the regression of each variable Xj on all other respective variables Xl, l = 1, …, 20, l ≠ j (see S2 Fig). Moreover, five out of ten predictors have univariable effects that are larger than their multivariable effects. (Here, multivariable effects are obtained by fitting a model with all predictors and noise variables).

 3.2.3 Simulation of outcome <italic>Y</italic>

The outcome Y is simulated as follows:

For linear regression: Y = xβ + ϵ, with ϵ ∼ N(0, σ2), and σ2 chosen such that R2 = 0.45 (setting 1, main scenario), R2 = 0.15 (setting 2, low R2 scenario), or R2 = 0.7 (setting 3, high R2 scenario). The intercept β0 is set to 36. The vector x = (1, x1, …, x10) denotes a simulated realization of the variables X1, …, X10 with an added constant for the intercept.

The required σ2 values for obtaining R2 = 0.45, R2 = 0.15, or R2 = 0.7 can be calculated as follows [9]: σ 2 = V a r ( X P β ) 1 - R 2 R 2 , where Var(XPβ) is calculated with the design matrix XP obtained from the approximate “population dataset” DP.

For logistic regression: outcomes Y are drawn from a Bernoulli distribution with event probability 1 1 + exp ( - c x β ) , with a constant c > 0.

First, we set c = 1 and adjust the intercept β0 manually such that the overall expected event probability equals either 0.3 or 0.05. The resulting Cox-Snell R C S 2 values are 0.40 for event rate 0.3 and 0.16 for event rate 0.05. These values constitute the main settings for logistic regression, but note that they are different from the R2 value of 0.45 in the main setting for linear regression (setting 1). Because setting 2 for linear regression considers 1/3 of the R2 value in setting 1, we add analogous “low R C S 2 settings” for logistic regression: c and β0 are adjusted to obtain 1/3 of the original R C S 2 values (0.40/3 = 0.13 for event rate 0.3, and 0.16/3 = 0.05 for event rate 0.05). In contrast to linear regression (setting 3), we do not include an additional high R2 setting: the maximum Cox-Snell R2 values that are possible in theory are less than 1 (for event rate 0.3: approx. 0.71, for event rate 0.05: approx. 0.33), thus the R2 values in the main settings can already be considered as relatively high.

In summary, this yields the following settings 4–7, for which we also estimated the corresponding population areas under the receiver operating characteristic curve (AUC) based on the “population dataset” DP:

the overall expected event probability equals 0.3 with R C S 2 = 0 . 40, AUC = 0.90 (setting 4) or R C S 2 = 0 . 13, AUC = 0.73 (setting 5),

the overall expected event probability equals 0.05 with R C S 2 = 0 . 16, AUC = 0.94 (setting 6) or R C S 2 = 0 . 05, AUC = 0.78 (setting 7).

 3.2.4 Nonlinear functional forms

So far, we assumed that the functional forms of the effects of continuous predictors on Y are linear. In applied studies in biology and medicine, the actual functional forms of such variables might often be nonlinear, but researchers nonetheless fit a model with linear functional forms, e.g., because they are not aware that some functional forms might be nonlinear, or because they prefer a simpler model. To analyze the behavior of variable selection methods in this scenario, we include settings 1b-7b (corresponding to settings 1–7) where all predictors have nonlinear functional forms. The models that we consider for analysing the simulated data (linear/logistic regression) will not take the nonlinear functional forms into account and will thus be misspecified.

For each continuous predictor Xj, we define a function gj(x) that describes the nonlinear functional form of the effect of the predictor on Y. We choose various functional forms: quadratic, log-quadratic, exponential and sigmoid. The functions are depicted in S3 Fig; exact definitions are given in S1 Appendix.

The nonlinear composite predictor is then simulated as β 0 ( g ) + β 1 ( g ) g 1 ( X 1 ) + … + β 10 ( g ) g 10 ( X 10 )

Here, the coefficients β j ( g ) (the letter g alludes to the nonlinear transformations gj) are chosen for continuous predictors such that β j ( g ) S D ( g j ( X j ) ) = ! | β j s d | = | β j | S D ( X j ) , to obtain effects that are comparable in magnitude to the linear effects. The standard deviations SD(Xj), SD(gj(Xj)) are calculated based on the approximate “population dataset” DP.

After determining β(g), the outcome Y is simulated as previously described in Section 3.2.3, with xβ replaced by the nonlinear composite predictor.

The simulation settings 1–7 with linear effects and settings 1b-7b with nonlinear effects are summarized in Table 3. For settings 1b-7b, the R2 values hold for the true model with nonlinear functional forms; the achieved R2 values when functional forms are misspecified are expected to be lower. For logistic regression, using the coefficients β j ( g ) for nonlinear effects in the main scenario yields slightly larger Cox-Snell R C S 2 values compared to the analogous settings with linear effects: 0.43 for event rate 0.3 and 0.20 for event rate 0.05. For the low R C S 2 scenario, c and β 0 ( g ) are adjusted to obtain 1/3 of the R C S 2 values (0.43/3 = 0.14 for event rate 0.3, and 0.20/3 = 0.07 for event rate 0.05), analogously to the procedure for settings with linear effects.

10.1371/journal.pone.0308543.t003 Overview of simulation settings.
linear regression logistic regression
event rate 0.3 event rate 0.05
main scenario…
…with linear effects setting 1 setting 4 setting 6
R2 = 0.45 R C S 2 = 0 . 40 R C S 2 = 0 . 16
…with nonlinear effects setting 1b setting 4b setting 6b
R2 = 0.45 R C S 2 = 0 . 43 R C S 2 = 0 . 20
low R2 scenario…
…with linear effects setting 2 setting 5 setting 7
R2 = 0.15 R C S 2 = 0 . 13 R C S 2 = 0 . 05
…with nonlinear effects setting 2b setting 5b setting 7b
R2 = 0.15 R C S 2 = 0 . 14 R C S 2 = 0 . 07
high R2 scenario…
…with linear effects setting 3
R2 = 0.7
…with nonlinear effects setting 3b
R2 = 0.7

For the global model in the settings with nonlinear effects, we will not only calculate the usual standard errors of the regression coefficients, but also robust standard errors [31], to check whether robust SEs improve the coverage of the confidence intervals. If robust SEs improve the coverage for the global model, it would be interesting to analyze whether this is also the case for models obtained by variable selection; however, combining robust standard errors with variable selection requires some further work and would go beyond the scope of the proposed study. For now, we will restrict the investigation of robust SEs to the global model for linear regression.

 3.2.5 Simplified settings

While our main focus is on simulating variables of various distribution types (e.g., Bernoulli, normal, and log-normal) and with correlation matrix Σ based on the empirical correlation matrix from the NHANES data (S1 Table), we are also interested in the behavior of the variable selection methods for data with simpler distribution-correlation structures. We thus consider the three following simplified scenarios:

The variables are multivariate normal and independent: X ∼ N 20 ( 0 , I 20 ), with I20 denoting the 20 × 20 identity matrix.

The variables are multivariate normal and correlated: X ∼ N 20 ( 0 , Σ ), with Σ denoting the correlation matrix as described above.

The variables have the same individual distributions as described in Section 3.2.1 (Fig 1), but are not correlated.

For each of these three scenarios, we will consider the settings 1–2 and 4–7 (i.e., the main scenario and low R2 scenario) with linear effects. This yields 3 * 6 = 18 simplified settings. For logistic regression, using the coefficients β j s d from Section 3.2.2. will yield Cox-Snell R C S 2 values that are slightly different from those given in Table 3.

Depending on the results for settings 1b-2b and 4b-7b with nonlinear effects, we might additionally consider nonlinear effects for the simplified scenario 3 (variables not multivariate normal and not correlated).

 3.2.6 Sample sizes

For linear regression, we consider eight different sample sizes: 100, 200, 400, 500, 800, 1600, 3200, and 6400. These sample sizes result when doubling sample size six times from 100. Additionally, the sample size 500 is included because it corresponds to EPV = 25, and this EPV value was specifically mentioned in the recommendations of Heinze et al. [1].

For logistic regression, we first choose sample sizes corresponding to EPV = 25: n = 1667 (event rate 0.3) respectively n = 10, 000 (event rate 0.05). The other sample sizes for logistic regression are chosen differently depending on the event rate. For event rate 0.3, sample sizes are “aligned” to the samples sizes { n 1 ( l i n ) , … , n 7 ( l i n ) } = { 100 , 200 , 400 , 800 , 1600 , 3200 , 6400 } of linear regression as follows: sample sizes n k ( l o g ) for logistic regression are chosen such that at sample size n = n k ( l o g ), the regression coefficients in the logistic regression have approximately the same standard errors as the regression coefficients in the linear regression at n = n k ( l i n ). Our procedure for aligning the sample sizes is described in detail in S1 Appendix.

Because this procedure is unstable for small event rates, we do not use the alignment based on standard errors for event rate 0.05. Instead, we choose sample sizes corresponding to the EPV values in linear regression.

The resulting sample sizes are displayed in Table 4. The numbers below the sample sizes indicate the corresponding EPV values. For event rate 0.05, we will first include sample sizes only up to 10,000 (EPV = 25) to save computation time. We expect the variable selection methods to behave similarly for both event rates (0.3 and 0.05). If we observe different behaviors for event rate 0.05, we will include the additional sample sizes.

10.1371/journal.pone.0308543.t004 Sample sizes and EPV values for linear and logistic regression.
linear regression n 100 200 400 500 800 1600 3200 6400
EPV 5 10 20 25 40 80 160 320
logistic regression, event rate 0.3 n 183 365 730 1667 1461 2922 5844 11,687
EPV 2.75 5.48 10.95 25 21.92 43.83 87.66 175.31
logistic regression, event rate 0.05 n 2000 4000 8000 10,000 – – – –
EPV 5 10 20 25 – – – –

In S1 Appendix, we additionally report expected shrinkage factors for each setting, based on sample size and R2 [32, 33].

 3.3 Estimands and other targets (E)

As estimands, we consider the true regression coefficients of the data generating models. As further targets, we are interested in model selection (e.g., whether the true model is selected) and predictive performance of the selected models.

For the settings with linear functional forms, the regression coefficient estimands are the coefficients β (respectively c β for logistic regression) as described in Sections 3.2.2 and 3.2.3. For the settings with nonlinear effects, we cannot take the coefficients β(g) as defined in 3.2.4 as estimands, because our linear/logistic regression models will not take nonlinear functional forms into account and will thus be misspecified.

Instead, we consider two alternative versions of estimands based on two different linear approximations of the nonlinear functions. Recall that we simulate the nonlinear composite predictor as β 0 ( g ) + β 1 ( g ) g 1 ( X 1 ) + … + β 10 ( g ) g 10 ( X 10 ) .

Our first option for the estimands is β(proj): β 0 ( g ) + β 1 ( g ) g 1 ( X 1 ) + … + β 10 ( g ) g 10 ( X 10 )≈ β 0 ( p r o j ) + β 1 ( p r o j ) X 1 + … + β 10 ( p r o j ) X 10 , where β(proj) are the coefficients obtained by projecting the true model with nonlinear functional forms onto one with linear functional forms. This projection is approximated by using the dataset DP as a “surrogate” for the population and fitting a linear/logistic regression model with linear functional forms to the nonlinear composite predictor (for linear regression) or to the outcome Y that was simulated based on nonlinear functional forms (for logistic regression).

As the second option, we consider β(AS): β 0 ( g ) + β 1 ( g ) g 1 ( X 1 ) + … + β 10 ( g ) g 10 ( X 10 )≈ β 0 ( A S ) + β 1 ( A S ) X 1 + … + β 10 ( A S ) X 10 , where β(AS) are the “average slope” coefficients. In contrast to the “projected” regression coefficients, here each variable is considered individually. Each nonlinear effect gj(x) is approximated as αjx, where αj is the average slope of gj weighted by the density of the j-th predictor Xj.

More precisely, let fj be the density function of Xj as estimated from the NHANES data (i.e., the density function that is used as input for the simulation, see Section 3.2.1). We aim to approximate the “average slope” integral ∫ m i n ( X j ) m a x ( X j ) g j ′ ( x ) f j ( x ) d x

For this purpose, we construct a partition x 1 j = min ( X j ) ≤ … ≤ x 1001 j = max ( X j ) of the range of Xj with equal sub-interval lengths d j = x k + 1 j - x k j, where min(Xj), max(Xj) are obtained from the NHANES data. Then the integral is approximated by α j = ∑ k = 1 1000 d j g j ′ ( x k j ) f j ( x k j ) Finally, β(AS) is obtained by setting β j ( A S ) = β j ( g ) α j.

 3.4 Methods (M) 3.4.1 Overview of variable selection methods

We include the following methods:

Forward selection with AIC: starting from the model containing only the intercept, variables are iteratively added to the model based on their capability to decrease the AIC when included.

Stepwise forward selection with AIC (i.e., forward selection with backward elimination steps): like simple forward selection, this method starts from the intercept model and adds variables based on the AIC. However, in each step, re-exclusion of already selected variables is allowed, based on the capability to decrease the AIC when removed.

Backward elimination with α = 0.05, with BIC, with AIC, and with α = 0.5: starting from the global model, variables are iteratively removed, either based on their capability to decrease the BIC/AIC when removed, or based on the p-values of their coefficients. We do not consider a stepwise variant of backward elimination with forward selection steps, following the recommendations of Royston and Sauerbrei [28, p. 32] who argue that allowing re-inclusion of removed variables in backward elimination is rarely relevant, while allowing re-exclusion of included variables may cause a notable difference for forward selection.

Augmented backward elimination (ABE) with AIC [15]: backward elimination is combined with the change-in-estimate criterion [34, 35]. A variable that would be removed in backward elimination based on AIC may stay in the model if its removal would induce a large change in the estimated regression coefficients of the other variables that are currently in the model. As threshold for the standardized change-in-estimate, we choose τ = 0.05. We will use the R package abe [36].

Univariable selection with α = 0.05 and α = 0.20: a variable is selected if its regression coefficient in a univariable model is significant at level α. While many authors have advised against using univariable selection [5, 37, 38], the method is still often used in practice, which is why we include it in our simulation study.

Univariable selection with α = 0.20, followed by backward elimination with α = 0.05: frequently, researchers use this combination instead of using only univariable selection or only backward elimination [39, 40] However, the warnings against univariable selection still apply to the combination method.

Lasso [16]: a penalty on the coefficients is added to the OLS criterion (linear regression) or the negative log-likelihood (logistic regression), causing shrinkage of the coefficients toward zero and setting some of them to exactly zero.

Relaxed Lasso [9, 17]: variables are selected with the Lasso, but the shrinkage of the coefficients of the selected variables is relaxed by refitting the model with the selected variables without penalty.

Adaptive Lasso [18]: first, the global linear/logistic model is fit, then a Lasso with variable-specific weights for the penalty is estimated. The estimates from the first step serve to get the variable-specific weights for the second step: the weights are calculated such that a variable with larger regression coefficient in the first step is penalized less than a variable with smaller regression coefficient.

For all variants of the Lasso, we will use the R package glmnet [41]. The complexity parameter λ will be tuned with 10-fold cross-validation (CV). As performance criterion for the prediction on test sets during CV, we use the mean squared error for linear regression and deviance for logistic regression. For the relaxed Lasso, we additionally consider tuning λ with the BIC.

We also consider the global model with all variables.

 3.4.2 Firth correction in logistic regression

In the models for logistic regression, separation may occur (i.e., perfect separation of events and non-events by a linear combination of covariates), particularly for small to medium sample sizes and low event rates [42]. In this case, at least one parameter estimate is infinite. While separation can be detected by linear programming [43], we found that in practice, a simple and robust check can be performed by inspecting the model standard errors of the regression coefficients. If at least one standard error is extremely large, this indicates separation. A possible solution to the problem of separation is to apply the Firth correction to obtain finite parameter estimates [42, 44].

In the simulation settings for logistic regression, we check for each individual simulated dataset whether separation occurs. In the case of separation, we apply the Firth correction (with the FLIC intercept correction [45] to obtain unbiased predictions), otherwise we use the standard logistic regression. When Firth correction is applied, confidence intervals for the regression coefficients are calculated based on the profile penalized likelihood, otherwise based on the profile likelihood.

We describe our procedure to check for separation based on the model standard errors of the coefficients in S1 Appendix.

 3.5 Performance measures (P)

We organize the performance measures into three categories, based on which estimands/targets they pertain to. Formulas for all performance measures are given in S1 Appendix.

Performance measures for the regression coefficients as estimands include bias and R M S E · n (root of expected mean squared error multiplied by n) of the estimated regression coefficients, and coverage and w i d t h · n of the 95% confidence intervals of the coefficients. (The RMSE of coefficients and width of confidence intervals are multiplied by n for better comparability across sample sizes.) Moreover, we consider the power for predictors and the type 1 error for noise variables (i.e., whether the confidence interval for the respective regression coefficient contains zero), as well as the selection rates of the variables (i.e., whether the regression coefficients are zero): the true positive rate for predictors and the false positive rate for noise variables. We also include Kendall’s τB [46] to measure the agreement of the estimated ranking of variables (defined by ordering the variables based on absolute values of the estimated standardized regression coefficients) with the “true” ranking of the variables (defined by ordering the variables based on absolute values of the true standardized regression coefficients).

For bias and RMSE · n of coefficients, coverage and width · n of confidence intervals, and type 1 error/power for variables, the calculation can be performed unconditionally or conditionally on selection. In the unconditional approach, the coefficients and their confidence limits for non-selected variables are set to zero, while the conditional approach includes only simulation runs where the specific variable is selected.

Performance measures for model selection as target include the selection rate of the true model consisting exactly of the ten predictors, the selection rate of an “over-selection” model which we define as a model including all predictors as well as at least one noise variable (previously called an “inflated” model [15]), and the selection rate of any “under-selection” model defined as a model not containing all predictors but possibly including noise variables (previously called a “biased” model [15]).

Finally, we use multiple performance measures for prediction. Predictive performance is evaluated on a large test dataset (ntest = 10, 000). One test dataset is simulated for each simulation setting. Prediction is assessed locally, i.e., at each value of the true linear predictor (local bias and local R M S E · n), as well as globally with the global R M S E · n (i.e., the root of the expected mean squared error of the estimated vs. true linear predictor multiplied by n) and the global MAE (i.e., the expected median absolute error of the estimated vs. true linear predictor). For logistic regression, global predictive performance is additionally evaluated with the AUC, i.e., area under the receiver operating characteristic curve. For both linear and logistic regression, the calibration of the predictions is measured with the integrated calibration index (ICI) [47]. The ICI is defined as the mean distance of the predicted outcomes/probabilities to the corresponding points on the calibration curve.

The performance measures for the regression coefficients and for model selection are primarily relevant for descriptive models, while performance measures for predictive performance are mainly relevant for prediction models. However, a descriptive model may also be suitable for prediction; therefore, performance measures for prediction could also be relevant for descriptive modeling. Vice versa, in prediction models, aspects such as interpretability, fairness etc. often play an important role; researchers might thus consider performance measures such as bias of coefficients also for prediction models.

 3.6 Monte Carlo errors and number of simulation runs

The number of simulation repetitions nsim must be chosen large enough to estimate the performance measures with suitable accuracy, i.e., the Monte Carlo errors of the measures must be acceptable. We use the coverage of the confidence intervals as reference measure. The Monte Carlo standard error for the coverage can be calculated with the formula c o v e r ^ ( 1 - c o v e r ^ ) n s i m , where c o v e r ^ is the coverage estimated via simulation [19]. If nsim = 2000 and c o v e r ^ = 95 %, the Monte Carlo SE is about 0.5%. If nsim = 2000 and c o v e r ^ = 50 % (the worst-case scenario), the Monte Carlo SE is about 1%, which is still acceptable. Therefore, we plan to use nsim = 2000 in our simulation for all settings (provided that this will be computationally feasible). We will then calculate the Monte Carlo SEs for all performance measures.

 4 Code review

To ensure reproducibility, as well as readability, the code will be checked by another researcher (a “code reviewer”) who works at the same institute as the first, second and last author of this protocol, but was not involved in planning the study. After writing the code, the first author (T.U.) will hand over the code to the code reviewer, together with instructions for running the code as well as some partial results (using less than the full nsim = 2000 repetitions). The code reviewer will then check the plausibility of the partial results and provide feedback on the simulation code, focusing on a) data generation, b) the implementation of the compared models, and c) the implementation of the performance measures applied to these models. Once T.U. and the code reviewer have agreed upon a final version of the code, T.U. will re-run the partial results, and the code reviewer will check the complete computational reproducibility by re-running the code on another machine. This check for reproducibility is done on the partial results as the generation of the final results is expected to require large amounts of computational resources. Once the reproducibility check has successfully concluded, T.U. will perform the full nsim = 2000 repetitions to generate the final results.

 5 Final remarks

Our simulation study will enable researchers to better understand the consequences of variable selection, and will clarify differences in the performance of different selection methods depending on the considered scenarios. To make the results of the study more accessible and interpretable, we plan to display all results in an interactive web app (Shiny app) that will be published alongside the main paper. We will also make our code available on a Git repository, and will specify random seeds to ensure reproducibility of the results.

The performance measures for our study (Section 3.5) are defined as expected values and probabilities. Their estimation by simulation thus always involves taking the mean over (a part of) the simulation repetitions. However, if one only calculates the mean over the repetitions, one might miss relevant properties of the distribution of the values over the simulation repetitions. We will thus use distribution plots and correlation analyses to evaluate the simulation results in more detail [19]. Moreover, we will analyze how many variables were selected by each variable selection method. We did not include model size as a performance measure in Section 3.5 because there is no clear target value and smaller/larger values are not automatically better/worse (a smaller model size is preferable in some applications, but might be less relevant in others). A specific focus on model size (e.g., comparing different variable selection methods under constraints w.r.t. the number of chosen variables) would require a different study design.

Multicollinearity is an important topic in the context of variable selection. Data-driven variable selection methods tend to perform worse if there is a high degree of correlation between the predictors, and their performance will improve the less the predictors are correlated with each other. Before regression analysis is performed in an applied study, the correlations between the independent variables should be checked during initial data analysis [48]. For our simulation study, we have carefully chosen true correlations between the independent variables based on a real correlation matrix from NHANES data. As mentioned in Section 3.2.1, S2 Fig shows the coefficients of determination R j 2 for the regression of each variable Xj on all other respective variables Xl, l = 1, …, 20, l ≠ j. The R j 2 values range from 0 to 0.56, demonstrating differing degrees of dependence between the predictors. These values will be considered when interpreting the simulation results.

In future work, it would be interesting to consider various extensions of our simulation. For example, while we focus on linear and logistic regression in the present protocol, data-driven variable selection is also often used in the context of survival analysis. We plan to conduct a further simulation study comparing different data-driven variable selection methods for Cox regression and the accelerated failure time model.

In the present study, we include several settings where all predictors have true nonlinear functional forms, but we nevertheless fit all models with linear functional forms; this mimics the frequent misspecification of models in practice. Generally, when fitting a regression model with linear effects, it is advisable to check for misspecification by analyzing the residuals. If misspecification is only mild, then a model with linear effects might still be justifiable. If misspecification is too severe, functional form selection can be performed to account for nonlinear effects, e.g., with spline-based approaches. In future work, our study could be extended by considering the combination of variable selection and functional form selection, which is a complex issue [39].

We focus on low-dimensional data in our study. Future studies could compare variable selection methods for high-dimensional data. Finally, our study considers variable selection in a frequentist framework. Future simulation studies could also evaluate Bayesian methods for variable selection.

 Supporting information Correlation network graph.

(PDF)

 Absolute standardized regression coefficients plotted against coefficients of determination for each independent variable.

(PDF)

 Nonlinear effects.

(PDF)

 Details of the simulation design.

(PDF)

 Correlation table.

(PDF)

We would like to thank the members of Topic Group (TG) 2 and the Publications Panel of the STRengthening Analytical Thinking for Observational Studies (STRATOS) initiative for helpful comments. In particular, we thank Willi Sauerbrei, Frank Harrell, Nadja Klein and Harald Binder.

At the time of submission, STRATOS TG2 consisted of the following members (in alphabetical order): Michal Abrahamowicz, Harald Binder, Daniela Dunkler, Frank Harrell, Georg Heinze, Marc Henrion, Michael Kammer, Aris Perperoglou, Willi Sauerbrei, and Matthias Schmid. The group is co-chaired by Georg Heinze (georg.heinze@meduniwien.ac.at), Aris Perperoglou, and Willi Sauerbrei.

 References Heinze G, Wallisch C, Dunkler D. Variable selection–a review and recommendations for the practicing statistician. Biometrical Journal. 2018;60(3):431–449. doi: 10.1002/bimj.201700067 29292533 Sauerbrei W, Buchholz A, Boulesteix AL, Binder H. On stability issues in deriving multivariable regression models. Biometrical Journal. 2015;57(4):531–555. doi: 10.1002/bimj.201300222 25501529 Steyerberg EW, Eijkemans MJ, Habbema JDF. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of Clinical Epidemiology. 1999;52(10):935–942. doi: 10.1016/S0895-4356(99)00103-1 10513756 Hurvich CM, Tsai C. The impact of model selection on inference in linear regression. The American Statistician. 1990;44(3):214–217. doi: 10.1080/00031305.1990.10475722 Harrell FE. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Springer Series in Statistics. Cham: Springer International Publishing; 2015. Mickey RM, Greenland S. The impact of confounder selection criteria on effect estimation. American Journal of Epidemiology. 1989;129(1):125–137. doi: 10.1093/oxfordjournals.aje.a115101 2910056 Maldonado G, Greenland S. Simulation study of confounder-selection strategies. American Journal of Epidemiology. 1993;138(11):923–936. doi: 10.1093/oxfordjournals.aje.a116813 8256780 Derksen S, Keselman HJ. Backward, forward and stepwise automated subset selection algorithms: Frequency of obtaining authentic and noise variables. British Journal of Mathematical and Statistical Psychology. 1992;45(2):265–282. doi: 10.1111/j.2044-8317.1992.tb00992.x Hastie T, Tibshirani R, Tibshirani R. Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons. Statistical Science. 2020;35(4):579–592. doi: 10.1214/19-STS733 Hanke M, Dijkstra L, Foraita R, Didelez V. Variable selection in linear regression models: choosing the best subset is not always the best choice. Biometrical Journal. 2023. doi: 10.1002/bimj.202200209 37643390 Boulesteix AL, Lauer S, Eugster MJ. A plea for neutral comparison studies in computational sciences. PloS One. 2013;8(4):e61562. doi: 10.1371/journal.pone.0061562 23637855 Boulesteix AL, Wilson R, Hapfelmeier A. Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies. BMC Medical Research Methodology. 2017;17:138. doi: 10.1186/s12874-017-0417-2 28888225 Sauerbrei W, Abrahamowicz M, Altman DG, le Cessie S, Carpenter J, STRATOS initiative. STRengthening analytical thinking for observational studies: the STRATOS initiative. Statistics in Medicine. 2014;33(30):5413–5432. doi: 10.1002/sim.6265 25074480 Shmueli G. To Explain or to Predict? Statistical Science. 2010;25(3). doi: 10.1214/10-STS330 Dunkler D, Plischke M, Leffondré K, Heinze G. Augmented backward elimination: a pragmatic and purposeful way to develop statistical models. PloS One. 2014;9(11):e113677. doi: 10.1371/journal.pone.0113677 25415265 Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological). 1996;58(1):267–288. doi: 10.1111/j.2517-6161.1996.tb02080.x Meinshausen N. Relaxed lasso. Computational Statistics & Data Analysis. 2007;52(1):374–393. doi: 10.1016/j.csda.2006.12.019 Zou H. The adaptive lasso and its oracle properties. Journal of the American Statistical Association. 2006;101(476):1418–1429. doi: 10.1198/016214506000000735 Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Statistics in Medicine. 2019;38(11):2074–2102. doi: 10.1002/sim.8086 30652356 Boulesteix AL, Groenwold RH, Abrahamowicz M, Binder H, Briel M, Hornung R, et al. Introduction to statistical simulations in health research. BMJ Open. 2020;10(12):e039921. doi: 10.1136/bmjopen-2020-039921 33318113 Centers for Disease Control and Prevention (CDC), National Center for Health Statistics (NCHS). National Health and Nutrition Examination Survey Data; 2023. Available from: https://www.cdc.gov/nchs/nhanes/. Sheppard JP, Stevens R, Gill P, Martin U, Godwin M, Hanley J, et al. Predicting Out-of-Office Blood Pressure in the Clinic (PROOF-BP): Derivation and Validation of a Tool to Improve the Accuracy of Blood Pressure Measurement in Clinical Practice. Hypertension. 2016;67(5):941–950. doi: 10.1161/HYPERTENSIONAHA.115.07108 27001299 Wynants L, Calster BV, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal [update 4]. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328 32265220 COVID-19 living review, summary details per model;. https://www.covprecise.org/living-review/ [Accessed: 2024-05-13]. Kipruto E, Sauerbrei W. Comparison of variable selection procedures and investigation of the role of shrinkage in linear regression-protocol of a simulation study in low-dimensional data. PloS One. 2022;17(10):e0271240. doi: 10.1371/journal.pone.0271240 36191290 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences. 2018;115(11):2600–2606. doi: 10.1073/pnas.1708274114 29531091 Pawel S, Kook L, Reeve K. Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method. Biometrical Journal. 2023. doi: 10.1002/bimj.202200091 36890629 Royston P, Sauerbrei W. Multivariable model-building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. John Wiley & Sons; 2008. Cario MC, Nelson BL. Modeling and generating random vectors with arbitrary marginal distributions and correlation matrix. Department of Industrial Engineering and Management, Northwestern University; 1997. Ghosh S, Henderson SG. Behavior of the NORTA method for correlated random vector generation as the dimension increases. ACM Transactions on Modeling and Computer Simulation (TOMACS). 2003;13(3):276–294. doi: 10.1145/937332.937336 White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica. 1980;48(4):817–838. doi: 10.2307/1912934 Riley RD, Snell KI, Ensor J, Burke DL, Harrell FE Jr, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: Part I–Continuous outcomes. Statistics in Medicine. 2019;38(7):1262–1275. doi: 10.1002/sim.7992 30347470 Riley RD, Snell KI, Ensor J, Burke DL, Harrell FE Jr, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II-binary and time-to-event outcomes. Statistics in Medicine. 2019;38(7):1276–1296. doi: 10.1002/sim.7992 30357870 Hosmer DW Jr, Lemeshow S. Applied logistic regression. New York: John Wiley & Sons; 2000. Bursac Z, Gauss CH, Williams DK, Hosmer DW. Purposeful selection of variables in logistic regression. Source Code for Biology and Medicine. 2008;3:17. doi: 10.1186/1751-0473-3-17 19087314 Blagus R. abe: Augmented Backward Elimination. R package version 5.1.1; 2022. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. Journal of Clinical Epidemiology. 1996;49(8):907–916. doi: 10.1016/0895-4356(96)00025-X 8699212 Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338. doi: 10.1136/bmj.b604 19336487 Sauerbrei W, Perperoglou A, Schmid M, Abrahamowicz M, Becher H, Binder H, et al. State of the art in selection of variables and functional forms in multivariable analysis—outstanding issues. Diagnostic and Prognostic Research. 2020;4(3). doi: 10.1186/s41512-020-00074-3 32266321 Mallett S, Royston P, Dutton S, Waters R, Altman DG. Reporting methods in studies developing prognostic models in cancer: a review. BMC Medicine. 2010;8:20. doi: 10.1186/1741-7015-8-20 20353578 Friedman J, Tibshirani R, Hastie T. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software. 2010;33(1):1–22. doi: 10.18637/jss.v033.i01 20808728 Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Statistics in Medicine. 2002;21(16):2409–2419. doi: 10.1002/sim.1047 12210625 Konis K. Linear programming algorithms for detecting separated data in binary logistic regression models [PhD thesis]. University of Oxford; 2007. Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27–38. doi: 10.1093/biomet/80.1.27 Puhr R, Heinze G, Nold M, Lusa L, Geroldinger A. Firth’s logistic regression with rare events: accurate effect estimates and predictions? Statistics in Medicine. 2017;36(14):2302–2317. doi: 10.1002/sim.7273 28295456 Kendall MG. The treatment of ties in ranking problems. Biometrika. 1945;33(3):239–251. doi: 10.1093/biomet/33.3.239 21006841 Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 2019;38(21):4051–4065. doi: 10.1002/sim.8281 31270850 Heinze G, Baillie M, Lusa L, Sauerbrei W, Schmidt CO, Harrell FE, et al. Regression without regrets—initial data analysis is an essential prerequisite to multivariable regression. 2023. doi: 10.21203/rs.3.rs-3580334/v1 10.1371/journal.pone.0308543.r001 Decision Letter 0 Tian Suyan Academic Editor 2024 Suyan Tian This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

16 Apr 2024

PONE-D-24-04044Evaluating variable selection methods for multivariable regression models: a simulation study protocolPLOS ONE

Dear Dr. Dunkler,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. As pointed out by the reviewer, the setting for the simulations is comparable simple compared to the real world cases, please address this issue seriously.Also, the authors mentioned when other relevant studies carried out a comparison between their proposed method and the existing methods, the number of the existing methods considered is very limited. However, the authors themselves have not carried out any comprehensive comparison with relevant methods. Please revise the corresponding paragraphs. Lastly, when I was a student and my professor told us that the coefficients before g(x) are still called linear coefficients, but the authors named them as nonlinear ones. Please double check it. If I am correct, please revise the corresponding equations and sentences.

Please submit your revised manuscript by May 31 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Suyan Tian

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. One of the noted authors is a group or consortium TG2 of the STRATOS initiative. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: No

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: No

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: No

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors consider variable selection in multivariate regression modeling from simulation perspective. Although the topic is very important in interesting, I am afraid the contribution does not reflect the importance and novelty. I have some specific comments for further consideration:

1- Variable selection first come to notice in (ultra)high-dimensional (HD) cases. In the simulations, I do not see this is highlighted and thus the simulation cannot be a reflective of real-world problems and not practical. In the revision, the authors must exactly specify the relation between the sample size and dimension.

2- In this paper, I did not see any trace of screening. Why did not authors consider selecting important variables based on the marginal correlation of conduct sure screening?

3- The problem of multicollinearity is not considered in variable selection which is absolutely important.

4- The measure R-squared is not a good measure of model fit in HD problems.

5- Form of explanatory variables does not play substantial role; however, the dependent variable plays. I do not see variant responses. Furthermore, in such studies, it is of essential need to consider additive structures in the predictive component. Further investigations for multivariate additive models are needed.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0308543.r002 Author response to Decision Letter 0 Submission Version 1

29 May 2024

All comments of the editor and the reviewer are addressed in the PDF file "Response to Reviewers".

Submitted filename: Response to reviewers.pdf

 10.1371/journal.pone.0308543.r003 Decision Letter 1 Tian Suyan Academic Editor 2024 Suyan Tian This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

26 Jul 2024

Evaluating variable selection methods for multivariable regression models: a simulation study protocol

PONE-D-24-04044R1

Dear Dr. Dunkler,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Suyan Tian

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I am happy with the revised version. The authors have clearly stated the underlying design of their simulation.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

 10.1371/journal.pone.0308543.r004 Acceptance letter Tian Suyan Academic Editor 2024 Suyan Tian This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

31 Jul 2024

PONE-D-24-04044R1

PLOS ONE

Dear Dr. Dunkler,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Suyan Tian

Academic Editor

PLOS ONE