Consider a data set consisting of two numeric matrices, X_n×p and W_n×q, holding data for n individuals (rows) and p (X) and q (W) features in columns, respectively. For instance, X may be a matrix with genotype codes at p SNPs and W may be a matrix providing GE levels assessed at q genes. The columns of X = {x₁,x₂,…,x_p} and of W = {w₁,w₂,…,w_q} can be viewed as axes spanning two linear spaces (L_X and L_W, respectively). The linear spans (extensions to nonlinear settings will be addressed in the discussion) of X and W consist of all the vectors that can be obtained by forming linear combinations of the columns of each of these sets, that is LX={xs:xs=Xαs=∑j=1pxjαsj} and LW={ws:ws=Wδs=∑j=1qwjδsj}, for all real-valued vectors α_s = {α_s1,…,α_sp} and δ_s = {δ_s1,…,δ_sq}. In the following, we will use W as the input set and X as the output set; however, the methods proposed are not symmetric and additional knowledge can be gained by switching the roles of X and W.

For each vector x_s∈L_X, one can estimate the proportion of variance that can be explained by linear regression on L_W using a model of the form xs=Wβ+ε. (1)

For cases where q is large, the proportion of variance of x_s that can be explained by regression on L_X (Rxs2) can be estimated by regarding both β and ε as Gaussian independent random variables, βiid∼N(0,σβ2) and, εiid∼N(0,σε2). Upon appropriate scaling of the columns of X (see Materials and Methods for details) Rxs2=σβ2σβ2+σε2 can be interpreted as the proportion of variance of x_s that could be explained by regression on the features included in W. The variance parameters involved (σβ2 and σε2) can be estimated using Bayesian or Likelihood methods (e.g., restricted maximum likelihood, REML, [7]), as such methods were designed to handle common challenges of overfitting and collinearity in high dimensional data.

In the preceding paragraph we describe how one can estimate the proportion of variance of a vector in L_X (x_s) that can be explained by regression on W. Next, we generalize the idea to all vectors in L_X. However, L_X contains an infinite number of vectors; therefore, some approximation is needed. Perhaps the most natural approach for estimating the proportion of variance of vectors in L_X that can be explained by regression on L_W is to regress each of the columns of X on W. Such an analysis would produce a sequence of R² estimates {Rx12,Rx22,…,Rxp2}, and the average R², RX∼W2=p−1∑s=1pRxs2, could be used to estimate the overall proportion of variance of X that could be explained by regression on W. However, one limitation of this approach is that the columns of X are not necessarily independent. Many features may cluster (e.g., genes may be co-expressed, or SNPs may be in high linkage-disequilibrium) leading to groups of highly unbalanced sizes. When some features are highly-correlated, the simple average of individual R²-values may be driven by a few clusters of the columns of X. Furthermore, when X is ultra-high dimensional (e.g., hundreds of thousands or million features) estimating Rxs2 (s = 1,…,p) one-feature-at-a-time will be computationally challenging. Therefore, to address these problems, we propose two methods that use independent vectors from the span of the output set; each of these methods are explained next.

Since L_X is infinite, one cannot estimate Rxs2 for all vectors in L_X. However, one can ‘explore’ the linear span of the output set by generating random vectors in L_X of the form x_s = Xα_s, where α_s is sampled from some distribution. This can be repeated for a large number of vectors in L_X to produce a sequence of estimates {Rxs2}, and the resulting sequence can be used to estimate the average proportion of variance explained as well as other features of the distribution of the sequence. The method is summarized in Box 1. Importantly, if α_s and α_s′ are independent, so are x_s and x_s′. Indeed, noting that X is not random and assuming that α_s and α_s′ are sampled independently, we have that p(Xαs,Xαs′)=p(Xαs|Xαs′)p(Xαs′)=p(Xαs)p(Xαs′|X); therefore, x_s and x_s′ are independent.

In Box 1 we did not specify how the α_s are generated; for this aspect of the algorihtm there are countless possibilities: weights can be sampled from distributions with continuous support (e.g., p-variate Gaussian) or from mixture models with a point of mass at zero. The weights may be independent or correlated, and the distributions may be symmetric or skewed. We will show later on (using data from chicken genomes) that the process used to generate the weights may affect some features of the distribution of the R² values, albeit not necessarily the mean or the median R². The possibility of using different processes for generating random vectors in L_X gives the MC-ANOVA a great deal of flexibility. For example, this method could be used to assess how the distribution of the proportion of variance explained may change for different trait architectures–we will further explore that flexibility in greater detail in one of the case studies presented below.

An orthogonal basis for the row-space of X can be obtained from the singular-value decomposition of X=UXDXVX′, where U_X and V_X are the left- and right-singular vectors of X respectively, and D_X is a diagonal matrix with the singular values of X in the diagonal. Both U_X and V_X are orthonormal, thus UX′UX=I and VX′VX=I. Each vector in L_X can be represented as a linear combination of the left-singular vectors of X. Therefore, our second method estimates the proportion of variance of each of the left-singular vectors of X that can be explained by regression on W, and produces a global R² estimate using a weighted sum of the R² estimated for each singular vector (Box 2, note that di2 and U_X in Box 2 are also the non-zero eigenvalues and the eigenvectors of XX′, respectively).

We evaluated the statistical performance of the two methods described above using simulated data panels with a known proportion of variance shared between input and output data set. We also compared the performance of the two proposed methods with that of the Partial Least Squares (PLS, [3])–a method commonly used to analyze high dimensional data. We considered two simulation settings. In both cases, the input set was obtained from a wheat (Triticum) genotype data set generated by the International Maize and Wheat Improvement Center (CIMMYT) which contains genotypes at 1,279 DNA-markers assessed in 599 wheat inbred lines (see Materials and Methods for further details on this data set).

We note here that while a genotype matrix contains strictly discrete values (0/1 or -1/1 for inbreed lines and 0/1/2 or -1/0/2 for outbred diploid individuals) the linear span of it includes vectors in Rⁿ. The vectors in the linear span of the genotypes can be thought as ‘breeding values’ formed as linear combinations of genotypes.

In our first simulation setting, W_599×1,279 was the genotype matrix and X_599×1,279 was obtained by adding iid (independent and identically distributed) Gaussian noise to the genotype matrix. We tuned the variance of the noise to generate scenarios of the proportion of variance of X explained by W ranging from 0 (X was pure noise) to 1 (X = W). For each simulated data set we then estimated the proportion of variance of X explained by regression of W using random-effects models, with variance parameters estimated using REML [7] (see Materials and Methods for details).

The Monte Carlo method estimated the proportion of variance of X explained by W without any noticeable bias (Table 1). However, the regression of the left-singular vectors of X on W in Eigen-ANOVA produced estimates that were downwardly biased in case the true proportion of variance of X explained by W was large (e.g., >0.5). Further inspection of the results for individual MC replicates suggested that the bias of the Eigen-ANOVA method was likely due to a relatively large number of ‘corner’ solutions (zero estimated proportion of variance) which were common for high-order eigenvectors (i.e., those with small eigenvalue)–we illustrate this in an analysis of multi-omic cancer data further below. The use of PLS led to a downwardly bias estimates in cases where the proportion of variance of X explained by W was low (0.1, 0.3, 0.5) and an upwardly biased estimates when proportion of variance explained was high (0.8, 0.9).

We then considered a second simulation setting to contemplate cases involving asymmetric proportion of variance explained. To achieve this, we formed X using a subset of the wheat marker genotypes (5%, 10%, 30%, 50%, 80%, 90%, 95%) and formed W by binding the columns of X with additional columns filled with iid Gaussian noise (Z), that is W_599×1,279 = [X_599×p, Z_{599×(1,279−p)}] (p<1,279). The columns of X and W were centered and scaled to unit variance; therefore the share of the variance of W explained by X (RW∼X2) is known and equal to, p1,279. Similarly, the proportion of variance of X explained by W (RX∼W2) is 1 because X is included in W.

In our second simulation study, the MC-ANOVA method rendered nearly unbiased estimates of the proportion of variance of one set explained by the other (Table 2). However, the Eigen-ANOVA method and the PLS produced noticeable biases, with Eigen-ANOVA method again generating downwardly biased estimates in cases where the true proportion of variance explained was high, and the PLS generating downwardly (upwardly) biased estimates whenever the true proportion of variance was low (high).

We used the MC-ANOVA and Eigen-ANOVA to quantify the proportion of variance explained in two experimental data sets. The first one contains a set of ultra-high-density (UHD) SNPs from chicken (Gallus gallus) genomes derived from a combination of whole-genome sequencing (WGS) and imputation. We used this data set to assess the proportion of variance of UHD genotypes that can be captured and predicted using low-density SNP sets. The second data set involved three omic-layers (gene expression [GE], methylation [ME], and copy-number-variants [CNVs]) of human (Homo sapiens) female breast cancer patients. We used this data set to assess the proportion of variance at one omic that can be explained by another omic.

The Eigen-ANOVA requires computing all the eigenvectors of the response matrix (say X) and then estimating proportion of variance of each of the eigenvectors explained by the explanatory matrix (e.g., W). The computational complexity of standard algorithms for singular-value decomposition is O(n³) (assuming n~p). On the other hand, the MC-ANOVA requires forming B random vectors fo the form x_s = Xα_s; ignoring the cost of sampling the weights, the computational complexity of forming each of this vectors is O(n²), again assuming n~p; thus, in general the MC-ANOVA will be computationally less involved as long as the number of vectors required for accurate estimation is smaller than n. In our experience, a few hundred random vectors (say 300) are enough to estimate the average, median, and SD of the R². Therefore, whenever the rank of the response matrix is high, the MC-ANOVA has clear computational advantages. These advantages would be particularly clear for very large rank matrices. Finally, we note that the estimation process of both Eigen-ANOVA and MC-ANOVA is 'embarrassingly' parallel since the R² of each of the vectors (either eigenvectors or random vectors) can be computed independently of each other.

Consistent with our simulation results, the analyses of experimental data showed that in problems involving a high R² the Eigen-ANOVA method yielded lower estimates of the proportion of variance explained than those obtained with the MC-ANOVA (e.g., see Fig 1 and Table 3). Inspection of the results of the Eigen-ANOVA for individual eigenvectors suggests that the downward bias of the method may originate from ‘corner’ solutions (zero-estimates of R²) for eigenvectors associated with small eigenvalues. Therefore, if the only goal is to estimate the proportion of variance of one set explained by another set, we recommend using the MC-ANOVA method.

The Eigen-ANOVA method yields R²-values for each of the eigenvectors of the output set. This information can help elucidate whether global patterns (e.g., those associated with the top-eigenvectors) in one information set can be predicted from information contained in another information set. For instance, our analysis of the multi-omic breast cancer revealed that the patterns described in the top-eigenvectors derived from GE and ME are very similar; therefore, one should not expect big differences in tumor classifications that are based on the top-eigenvectors derived from either set. Interestingly, we found that in the analyses of omic data the R² of individual eigenvectors showed a very sharp phase transition, suggesting that eigenvectors associated with intermediate and small eigenvalues may describe omic-specific patterns, or perhaps measurement error associated to each of the techniques.

The MC-ANOVA method can be used to characterize the distribution of the R² estimates across vectors in the linear span of the output set. We used this feature to study the effect of the trait-architecture on the distribution of the R² estimates. Our results indicate that while the average R² does not seem to be affected by the sparsity of the coefficients used to form random vectors (i.e., the α_s′), the dispersion and the shape of the distribution highly depend on the process used to generate the weights (Fig 2). Highly sparse weights lead to a distribution of the R² values that, compared with vectors that were less sparse, had higher dispersion and in some cases (e.g., when the proportion of variance explained was close to 1) was skewed (Fig 2).

An important feature of the methods proposed in this study is that the R² measure is not symmetric, in contrast to CCA. Our simulation study shows that if the underlying patterns are non-symmetric (e.g., when one of the linear spaces is a subspace of the other) the proposed estimation methods (in particular the MC-ANOVA) can detect the lack of symmetry very well (see Table 2). Interestingly, our analysis of multi-omic data from breast cancer patients exhibited cases where R² was rather symmetric (e.g., the regression ME~GE and the regression ME~GE) and others that were highly asymmetric (e.g., CNV~GE and GE~CNV). The asymmetric cases suggest that almost all the variability in CNV can be predicted from GE (and ME as well); however, only a fraction of the GE variance can be explained by differences in CNV patterns. This result is consistent with the hypothesis that most CNV have an impact on GE, but GE is also affected by factors other than CNV (e.g., methylation, environmental effects).

In this study, we focused on the application of the Eigen- and MC-ANOVA for problems involving two input sets (e.g., a low- and a high-dimensional SNP array, or two different omics) evaluated on the same set of individuals. However, with slight modifications, the MC-ANOVA method will be useful for evaluating the proportion of variance of vectors in the span of a training set (e.g., all the available genotypes/phenotypes) that could be captured/predicted by regression on a subset of it, e.g., founders, or “proven” individuals, e.g., [19, 20].

The methods discussed in this study are entirely based on linear models. However, both MC-ANOVA and Eigen-ANOVA can easily be extended to consider non-linear relationships by embedding each set using a non-linear mapping. For instance, in the case of SNPs, one could generate a linear space that accounts for additive and non-additive effects by considering contrasts for additive, dominance, and epistatic interactions [21]. More generally, one can consider embedding either X or W by transforming one or both sets using a non-linear mapping f(.) (e.g., Gaussian kernels). Then, the methods presented here could be applied using X˜=f(X) and W˜=f(W) as information sets within the context of Reproducing Kernel Hilbert Spaces regressions (e.g., [22, 23]).

In our applications, we considered one dependent and one explanatory set; however, the methodology presented in this study could be easily adapted to accommodate cases with one output set (e.g., Y) and multiple explanatory sets (e.g., X, and W). This can be done by expanding the model used to estimate R² [1] by including two random effects, each with its own variance parameter. Such methods could be used to answer potential questions such as what proportion of variance of gene expression may be explained by joint regression on methylation and copy-number-variants.

In conclusion, we developed two methods for estimating the proportion of variance explained in problems in which both the input and output sets are high-dimensional. The MC-ANOVA method provided nearly unbiased estimates across a range of simulation scenarios. In addition to providing estimates of the proportion of variance explained, the two methods can yield useful insight into the association patterns underlying multi-layered high-dimensional data.

The wheat data set used in Simulations 1 and 2 was generated by the International Maize and Wheat Improvement Center (CIMMYT). This data set provides genotypes at 1,279 molecular markers (Diversity Array Technology DNA markers) assessed in 599 wheat inbred lines. Further details about this data set can be found in [24]. The data set is available with the BGLR R-package [25].

The chicken data set used in Case Study 1 consisted of UHD SNP genotypes of 892 female and male chickens from six generations of a purebred commercial brown layer line of Lohmann Tierzucht GmbH. The genomes of 25 layers were sequenced at 8x read-depth. A total of 4.92M (M = million) SNPs were derived from these 25 genome sequences. The remaining layers (n = 867) were genotyped using the Affymetrix Axiom Chicken Genotyping Array [16] which contains ~600K (580,961) SNPs. Ni et al. [28] imputed the SNP-genotypes of those 867 layers to the whole-genome sequence (4.92 SNPs) using BEAGLE 3.3.2 [26] for phasing and MiniMac3 [27] for imputation. For details on the imputing procedure we refer to Ni et al. [28]. This produced a combined genotype file consisting of 4.92M SNPs from 892 = 25+867 genomes. We further filtered the combined genotype file by removing SNPs with minor-allele-frequency smaller than 0.005 (0.5%) and pruning adjacent SNPs that were in (almost) perfect LD (i.e., R² ≥0.99). A total of 1.79M SNPs passed these last two filters.

The breast cancer data set used in Case Study 2 was from the Cancer Genome Atlas (TCGA https://www.cancer.gov/tcga) and consisted of gene expression (GE), methylation (ME), and copy-number-variants (CNV) data from (n = 593) breast cancer tumors from female breast cancer patients.

Gene expression data (RNA-Sequencing counts) were generated using the Illumina HiSeq RNA V2 platform and DNA methylation profiles were determined using the Illumina HM450 platform. RNA-sequencing data were transformed using the natural logarithm and individual CpG site β-values were summarized at the CpG island level, using the maximum connectivity approach implemented in the WGCNA R package [29]. The CpG island summaries were transformed into M-values (M = β/(1-β) [30]). CNV profiles corresponded to gene-level copy number intensity derived from Affymetrix SNP Array 6.0 platform, using hg19 as reference.

From each of the three omics we removed features with a coefficient of variation smaller than 1% and those with a proportion of missing values greater than 20%. The missing values that remained were imputed using the clustering algorithm described in [31]. After imputation, each feature was adjusted for batch effects using ComBat [32]. After applying the steps described above, the data set used in the analyses consisted of the (log-transformed) expression of 20,319 genes, 11,552 CVN-sites, and ME intensity at 28,241 ME CpG islands.

For the MC-ANOVA and EIGEN-ANOVA methods, the proportion of variance of one set (e.g., X) explained by the other set (W) was estimated using a random-effects model of the form z=1μ+Wβ+ε, where z was either a random linear combination of the columns of X (MC-ANOVA, see Box 1) or one of the eigenvectors of XX′ (Eigen-ANOVA, see Box 2), μ is an intercept, β is a vector of Gaussian random effects, βiid∼N(0,σβ2), and ε is a vector containing error terms, which were also assumed to be Gaussian, εiid∼N(0,σε2). For computational convenience and without loss of generality, we reparametrized the above model in terms of a random-effects model, of the form z=1μ+u+ε, where u=Wβ∼MVN(0,WW′σβ2), where MVN() stands for Multivariate Normal Distribution. We centered and scaled the columns of W to a standard deviation equal to 1/ncol(W); this leads to a covariance structure, WW′ with an average diagonal equal to one; therefore, with this scaling, σβ2 can be interpreted as the amount of variance of z captured by regression on W and the ratio Rz2=σβ2σβ2+σε2 can be interpreted as the proportion of variance of z that can be explained by W.

We estimated the variance components of the above model using Restricted Maximum likelihood (REML [7]) which was implemented with a custom R-script that for optimization uses the bobyqa function of the minqa R-package [33]. The scripts used to fit variance components using REML are provided in the S1 File (see function fitREML).

We also estimated the proportion of variance of X explained by W (and the reciprocal when needed) by regressing X on W using the pls R-package [34].

The ability of the pls regression to fit X depends on the number of components used. To determine the number of components, we first fitted the PLS regressions with 1, 2, …, 100 components in 10-fold cross-validation and evaluated the cross-validation prediction mean-square error of each of the resulting models. We then selected the number of components that led to the smallest mean-squared prediction error and fitted a PLS regression with that number of components to the entire data set. The R² of the fitted model in the training data was used as an estimate of the proportion of variance of X that could be explained by W. The fitPLS function provided in the script provides a wrapper to the plsr function which implements the procedure described above.

Both simulations were implemented using genotypes from the wheat data set.