The RAINBOW model can be written as y=Xβ+Zcuc+Zriuri+ϵ, (1) where y is a n × 1 vector of phenotypic values, Xβ is a n × 1 vector of fixed effects including an intercept, a term to correct the population structure and other covariates, Z_cu_c and Zriuri are n × 1 vectors of random effects, and ϵ is a n × 1 vector of residual errors. Here β is a p × 1 vector of fixed effects, where p is the number of fixed effects. u_c and uri are m_c × 1 and mri×1 vector of genotypic values respectively, where m_c is the number of genotypes for additive polygenetic effects and mri is the number of genotypes for i-th SNP-set of interest. X, Z_c and Zri are n × p, n × m_c and n×mri design matrices that correspond to β, u_c and uri respectively. As the following formula Eq 2, we assume that the polygenetic effect u_c follows the multivariate normal distribution whose variance-covariance matrix is proportional to the additive numerator relationship matrix K_c. uc∼MVN(0,Kcσc2), (2) where σc2 is the additive genetic variance to be estimated in the “Estimation of variance components” section, and here m_c × m_c matrix K_c = A, where A is the known additive genetic relationship matrix estimated from marker genotype data W_c [37].

We also assume that the random effects from i-th SNP-set of interest uri follows the multivariate normal distribution whose variance-covariance matrix is proportional to the Gram matrix Kri. uri∼MVN(0,Kriσri2), (3) where σri2 is the genetic variance for i-th SNP-set to be estimated in the “Estimation of variance components” section, and Kri is the known mri×mri Gram matrix estimated from marker genotype data Wri belonging to the i-th SNP-set. We offer a linear, an exponential and a Gaussian kernel for the Gram matrix Kri, and faster computation can be realized for the linear kernel case (Supplementary Note in S1 Appendix) [24].

Finally, the residual term is assumed to identically and independently follow a normal distribution as shown in the following equation. ϵ∼MVN(0,Inσe2), (4) where I_n is a n × n identity matrix and σe2 is estimated in the “Estimation of variance components” section.

The variance components were estimated by maximum-likelihood (ML) [26, 38] and restricted maximum-likelihood (REML) [39]. Here we explain how to obtain ML and REML estimates of Eq 1 for the general Kri.

First we estimated the weights (we define w_c and wri) between the genetic variances (σc2 and σri2) by the following algorithm.

Setting initial parameters for w_c and wri: wc=wri=12. (5)

After estimating the weights w_c and wri, we estimated the variance components (σs2 and σe2) of the model Eqs 7 and 8 by EMMA/GEMMA using w^c and w^ri. Then we obtained σ^c2=w^cσ^s2 and σ^ri2=w^riσ^s2.

Our fitting method, as described above, is a two-step approach, which first estimates the weights of genetic variances, and then estimates the variance components of the model shown in Eqs 7 and 8 by EMMA/GEMMA with the estimated weights. On the other hand, some fitting methods that directly estimate the variance components for Eq 1 via AIREML (average information REML) [43] have also been proposed and implemented in some packages/software [44, 45]. The advantage of our two-step approach compared with the direct estimation approach via AIREML is that the search space of the weights is limited to the interval [0, 1], and the convergence is relatively warranted [46] even when the heritability is too low/high.

To test the significance of each SNP-set, we performed the LR test of whether σri2=0 or not. As a null hypothesis, the following model, which does not include the term of SNP-set effects was assumed. y=Xβ+Zcuc+ϵ. (12) In contrast, as an alternative hypothesis model, the multi-kernel linear mixed model (MKLMM) of Eq 1 was assumed. Therefore, we computed the following deviance after the estimation of variance components for each SNP-set. D=2×(l^R,model−l^R,null), (13) where l^R,model is the maximum of the restricted log likelihood for the model of Eq 1 and l^R,null is the maximum of the restricted log likelihood for the model of Eq 12.

Finally, we tested the significance of σri2 and calculate the p-value by assuming that the deviance in Eq 13 followed the mixture of two chi-square distributions with different degrees of freedom [47, 48]. D∼π0χ02+(1−π0)χ12, (14) where π₀ is the mixture parameter and here we used π₀ = 1/2.

In this study, 414 accessions of Oryza sativa subsp. indica were collected from “the 3,000 rice genomes project” (S1 Table) [7]. We used a marker genotype consisting of core SNPs defined by the Rice SNP-Seek Database as “404k CoreSNP Dataset”. Imputations were imputed using Beagle version 5.0 [49, 50]. We analyzed only bi-allelic sites over all accessions with a MAF ≥ 0.025 by using VCFtools version 0.1.15 [51]. In the following analysis, genotypes are represented as -1 (homozygous of the reference allele), 1 (homozygous of the alternative allele) or 0 (heterozygous of the reference and alternative alleles). As a result of this data processing, marker genotypes with 112,630 SNPs were used for the following simulation study.

To perform haplotype-based GWAS by regarding each haplotype block as a SNP-set, haplotype blocks were estimated from marker genotype data by using PLINK 1.9 [52–54]. As a result of estimation, we obtained 15,275 haplotype blocks consisting of 78,237 SNPs.

We considered two scenarios to validate our novel haplotype-based GWAS approach. In both models, phenotypic values were simulated as follows. y=X1β1+X2β2+X3β3+Zu+e, (15) where y is the vector of simulated phenotypic values of 414 accessions, X₁, X₂ and X₃ correspond to three quantitative trait nucleotides (QTNs) scored as -1, 0 or 1 (hereinafter, referred to as “QTN1”, “QTN2” and “QTN3” respectively), β₁, β₂ and β₃ are scalars representing the effects of the three QTNs, u is the vector of polygenetic effects, and e is the vector of the residuals.

Here, QTN1 and QTN2 were randomly selected from all genome-wide SNPs to satisfy that they belonged to the same haplotype block that harbored more than 4 SNPs. QTN3 was randomly selected from all the SNPs. We assumed that the effects of QTN1 and QTN2 had a variance 4 times greater than that of the effects of QTN3 to mainly check the detection power for the haplotype block. More details about the other terms are described in S1 Appendix.

The difference between two scenarios is based on the directions of the two QTN effects β₁ and β₂. Scenario 1 assumed that the directions of two effects were identical. That is, β1={β2(ρ12≥0)−β2(ρ12<0), (16) where ρ₁₂ is Pearson’s correlation coefficient between X₁ and X₂. We call this model as “coupling”.

Conversely, scenario 2 assumed that the directions of the two effects were opposite. That is, β1={−β2(ρ12≥0)β2(ρ12<0). (17)

We call this scenario 2 as “repulsion”.

To validate our novel approach, we compared the following four methods: a single-SNP GWAS [55], a haplotype-based GWAS introduced by Yano et al. (hereinafter, referred to as “HGF”) [28], the SKAT [15] as a SNP-set approach, and our novel approach, RAINBOW. For all methods, to account for the population structure, the two eigen vectors (which correspond to the top two eigen values) of the additive genetic relationship matrix were included in the model as fixed effects. The details of these four methods are described in S1 Appendix.

The value of −log₁₀(p) of each marker or haplotype block was calculated by the four GWAS methods 100 times for the two simulated scenarios, coupling and repulsion. In this study, the following summary statistics were used to evaluate the simulation results.

−log₁₀(p) and −log₁₀(p_a). The first summary statistic is −log₁₀(p) of each causal SNP or haplotype block itself. For haplotype-based GWAS methods, HGF, SKAT and RAINBOW, the significance of β₁ and β₂ was represented by −log₁₀(p) of the causal haplotype block to which X₁ and X₂ belong. In the single-SNP GWAS method, the −log₁₀(p) of β₁ and β₂ were calculated separately, even though these SNPs were in the same haplotype. To compare the single-SNP GWAS method with the haplotype-based GWAS methods, the −log₁₀(p) values were averaged over β₁ and β₂.

As some of these methods showed the results of inflated −log₁₀(p), we defined the following summary statistic to evaluate the degree of inflation. inflator=1L∑l=1L(−log10(pfalse,l)), (18) where p_false,l is the l^th p-values for false positives arranged in increasing order. In this study, L was set as 10. Then we adjusted −log₁₀(p) of the causal by using the inflator (Eq 18) as follows. −log10(pa)=−log10(p)−inflator, (19) where p_a is the p-value adjusted by the inflator.

Here, we calculated each summary statistic in two ways. The first method is to calculate each summary statistic by directly using −log₁₀(p) of each causal SNP / haplotype block. The other method is to calculate the summary statistics by regarding multiple SNPs or haplotype blocks within the extent of the LD as one set. In this study, we defined SNPs or haplotype blocks that satisfy the condition that they are within 300 kb from the focused SNP or haplotype block and the condition that their square of the correlation coefficients with the focused SNP or haplotype block are 0.35 or more as one set considering the LD. The highest value of −log₁₀(p) in the LD region was assumed to represent the values of the SNPs or haplotype blocks within the extent of the LD.

We calculated the recall, precision and F-measure means as other summary statistics to evaluate the GWAS results. These summary statistics were calculated from the numbers of SNPs or haplotype blocks that were true positives, false positives, false negatives and true negatives. Here, we regarded a SNP or haplotype block as “positive” when that SNP or haplotype block exceeded the threshold. In this study, the value of −log₁₀(p) so that the FDR (false discovery rate) was 0.01 was set as the threshold by using the Benjamini-Hochberg method [56, 57]. In addition, these three summary statistics, recall, precision and F-measure, were calculated by assuming that the highest value of −log₁₀(p) in the LD region represented the values of the SNPs or haplotype blocks within the extent of the LD.

Therefore, recall represents the proportion of causals detected by GWAS. In contrast, precision represents the ratio of the detected SNPs or haplotype blocks that were causals. Finally, F-measure was calculated as the harmonic mean of the recall and the precision, which evaluates the GWAS results comprehensively. The greater these three summary statistics, the better the results of GWAS are. Here we simply took the average of each summary statistic from all the 100 simulation results.

We calculated the mean of the AUC (area under the curve) for regions around the causals as a summary statistic. AUC refers to the area under the ROC (receiver operating characteristic) curve obtained by plotting the false positive rate on the horizontal axis and the true positive rate on the vertical axis when the threshold is varied. In this study, the AUC was calculated for the SNPs or haplotype blocks near the causal SNP / haplotype block (QTN1 and QTN2). In other words, the non-causal markers that had a strong LD with the causal SNP / haplotype block were regarded as false positives under this summary statistic. Therefore, this summary statistic indicates the extent to which the causal itself can be detected by GWAS without relying on the LD. Here, when taking the average of the AUCs obtained from the simulation results, two methods were used, either using all the 100 results or only using the results whose QTN1 and QTN2 were “detected”. Here, QTN1 and QTN2 were regarded as “detected” if −log₁₀(p_a) ≥ 1.5 for each method.