2287 genotypes were obtained from the Cancer Genomic Markers of Susceptibility (CGEMS) breast cancer study. The samples were sourced as described in [2]. Briefly, the samples comprised 1145 breast cancer cases and a comparable number (1142) of matched controls from the participants of the Nurses Health Study. All the participants were American women of European descent. The samples were genotyped against the Illumina 550K arrays, which assays over 550,000 SNPs across the genome. To assess the genetic identity shared between samples, we computed the fraction of SNPs with identical alleles for all possible pairs of individuals; none exceeded .

Additionally, 90 genotypes of American individuals of European descent (CEPH) and 90 genotypes of Yoruban individuals were obtained from the HapMap Project [3]. In both cases, the 90 individuals were members of 30 family trios comprising two unrelated parents and their offspring. SNPs in common with those assayed by the CGEMS study and located on chromosomes 1–22 were kept in the analysis (sex chromosomes were excluded since the CGEMS participants were uniformly female); a total of 481,482 SNPs met these criteria.

The method as described in [1] and summarized in the Introduction was implemented using R [4]. Subsets of the data described above were used to construct pools and , using the remaining genotypes as test samples for which the null hypothesis is true. A summary of the tests is provided in Table 1. In each test, SNPs which did not achieve a minor allele frequency in both and were excluded from the computation.

To explore the performance of the method in realistic situations, we carried out the computations described by Equations 1,2 for various , , and as described in Table 1. Distributions of for each of the tests described in Table 1 are shown in the corresponding figures listed in the table. We find that while the distributions of in-F, in-G and in-neither values of are distinct, calibrating thresholds for classifying an unknown sample is difficult without additional information. This is due to the fact that the distribution of for null samples deviates strongly from a standard normal in practice.

We begin first by considering a best-case situation in which and are both large samples of the same underlying population , and the samples to be classified are also from . Here, we randomly select 100 cases and 100 controls from CGEMS to form an out-of-pool test sample set comprising 200 individuals, using the remaining 1045 CGEMS cases and 1042 CGEMS controls as pools and , respectively. (Several such random subsets were created; the results were consistent and hence we present a single representative one.) (Equation 1, 2) was computed for all the samples and compared to a standard normal ( yields a nominal (p-value) of 0.05 and yields a nominal ). The sensitivity and specificities obtained are given in Table 2.

Distributions of values for all three groups of CGEMS samples are shown in Figure 1A. Notably, the distributions of in-F, in-G, and in-neither samples are all quite distinct. For the positive samples (those in or ), the classifier performs fairly well, correctly classifying 2083 samples (and calling 4 as in neither nor ). However, of the 200 test samples which were in neither nor , only 62 have ; the rate of false positives is thus 69% if is used as an indicator of group membership under the assumptions in [1] at the nominal (see Table 2).

Next, we consider a less ideal, yet probable, case in which the null samples are not from the same underlying population . Here, we leave and as above, and apply Equation 1, 2 to 90 HapMap American individuals of European descent (whom, one might assume, would be relatively similar to the Americans of European descent comprising groups and ). A plot of the value distributions is given in cyan in Figure 1B. Again, there is little overlap with the true positive distributions, but when comparing the values against , the sensitivity is quite low (see Table 2). A yet more extreme case, in which 90 HapMap Yoruban individuals were classified with respect to and , results in a distribution of values that overlaps with the values from group (Figure 1B, blue curve) and exceedingly low specificity (Table 2). We thus see in practice a strong dependence of upon the assumption that , , and are samples of the same population.

The reason for the high false-positive rates in practice despite the stringent nominal false positive rate is clear from the plots Figure 1A and 1B: namely, it can be seen that the putative null distribution (light grey line, , cf Equation 2) does not correspond to the observed distribution for samples for which the null hypothesis is correct, with differences in both the location and width.

The overall shift in the location of the distributions is a result of violations of the assumption that each sample , , and are drawn on from the same underlying population . The magnitude of this effect is derived in Text S1 as , where are the MAFs of the population from which is drawn (hence the different rightward shifts of the CGEMS, CEPH, and YRI distributions). Because of the large number of SNPs in Equation 2, small deviations from are magnified; even ancestrally similar populations, such as the 200 CGEMS test samples and the HapMap CEPHS, have different distributions of .

The broadening of the distribution is a result of correlation between SNPs. In Equation 2, it is assumed that the variance of the mean of be estimable by the mean of the variance, ie, , which is true for independent SNPs. However, if there exists average correlation amongst the SNPs (due to linkage disequilibrium),(4)which can be quite large even for small average correlation due to the high number of SNPs. The result of increased LD is a broader distribution of values, as observed in Figure 1A and 1B: we observe a narrower distribution of for the HapMap YRI samples versus the Caucasian CGEMS participants and HapMap CEPHs (the Yoruban individuals, who come from an older population, have lower average LD).

The effect of LD on the distribution of may be countered by selecting fewer SNPs; the results of this approach can be seen in Figure 1C and 1D and in Table 2. Here, 50,000 SNPs were selected, uniformly distributed across of the SNPs used in Figure 1A and 1B. 50,000 SNPs was shown in [1] to be a reasonable lower bound to detect at nominal one individual amongst 1000, which is the concentration of true positive individuals in this test. As is clear from Figure 1, reducing the number of SNPs narrows the distributions considerably, yet at the same time brings them closer together such that the crisp separation previously obtained is reduced. Using this method, we see that the 200 CGEMS samples now have a distribution closer to that of the putative null such that using a threshold of yields an improved—yet still larger than nominal—21% false-positive rate while maintaining a high 96.3% true positive rate. However, the misclassification rate is still over 50% for both HapMap samples, and improving these values requires compromising the sensitivity, a direct result of the overlapping distributions for the and HapMap samples.

Despite the low sensitivities obtained in our tests, it is apparent from Figure 1 that the true positive individuals have a significantly different distribution of values than do the null samples, such that if appropriate thresholds were selected the classification could be improved (note that in practice, the distributions of the true positive individuals are unknown, since reconstructing them requires full genotypes, not just the MAFs, of and ). One simple apprach, motivated by the observed separation of distributions in Figure 1, would be to collect a set of presumed-null genotypes from which to estimate the null distribution. Consider a situation in which we have and , along with an individual who is one of the 200 CGEMS samples not in or , but no other genotypes. We might reasonably turn to publicly available HapMap genotypes as a group from which to construct an empirical null distribution for setting thresholds. The lines in Figure 1A and 1C depict this case. Using the 0.05 and 0.95 quantiles obtained from the HapMap CEPH distribution (cyan bars) as thresholds improves the accuracy relative to using quantiles, but still incorrectly classifies half of the 200 CGEMS samples; the false positive rate is yet greater (and the true-positive rate smaller) when using the HapMap YRI quantiles (blue bars). Likewise, roughly a quarter of the HapMap CEPHs and the majority of HapMap YRIs lie outside the thresholds set from the 200 CGEMS samples in Figure 1B and 1D.

These examples, as well as the analytical results described in Text S1, show that deviations from the assumptions that , , and are i.i.d. samples of the same population can produce misleadingly large values of . While Equations 1, 2 produce good separation of the , and null sample distributions, appropriately calibrating the thresholds for classification is difficult in practice.

The stated purpose [1] of the method—namely, to positively identify the presence of a particular individual in a mixed pool of genetic data of unknown size and composition—is difficult to achieve. In this scenario, we have (from forensic evidence) and a suspect genotype . To apply the method, we would need 1) to assume that and are indeed i.i.d. samples of the same population ; 2) to obtain a sample which is also a sample of the underlying population , well-matched in size and composition to ; 3) to obtain an estimate of the sample size of such that sample-size effects can be appropriately discounted (see Text S1); and 4) to assume that the p-values at the selected classification thresholds are accurate. The high false-positive rates which result from even small violations of these criteria make it exceedingly likely that an innocent party will be wrongly identified as suspicious; it is even more likely for a relative of an individual whose DNA is present in .

Here the scenario of concern is that of a malefactor with the genotype of one (or many) individuals, and access to the case and control MAFs from published studies; could the malefactor use this method to discern whether one of the genotypes in his possession belongs to a GWAS subject? In this case, and are known to be samples of the same underlying population (due to the careful matching in GWAS), and their sample sizes are large and known. However, the malefactor still needs 1) to assure that is a member of this population as well (as shown by the poor results when HapMap samples were classified using CGEMS MAFs) and 2) to assume that the p-values at the selected classification thresholds are accurate. Additionally, the prior probability that any of the genotypes in the malefactor's possession comes from a GWAS subject is likely to be quite small, since GWAS samples are a tiny fraction of the population from which they are drawn. Even if the malefactor were able to narrow down the prior probability to one in three, a sensitivity of 99% and a specificity of 95% is needed to obtain a 90% posterior probability that the individual is truly a participant.

On the other hand, if the malefactor does have prior knowledge that the individual participated in a certain GWAS but does not know Y's case status, Equations 1, 2 permit the malefactor to discover with high accuracy which group was in. Additionally, in the case of a priori knowledge, the participant's genotype is not strictly necessary, since a relative's DNA will yield a large score that falls on the appropriate side of null.

Despite these limitations, we observe that the distributions of values for in-F, in-G, and null samples separate strongly, suggesting that each individual contributes a pattern of allele frequencies that remains in the pooled data. While calibrating thresholds to distinguish these distributions without additional information is not presently possible, the potential for more sophisticated methods to overcome these barriers cannot be discounted and presents an avenue for future work.

Moreover, we believe that the distance metric (Equations 1, 2) as presented may still have forensic and research utility. It is clear from both our studies and the original paper [1] that the sensitivity is quite high; in the (rare) case that a sample has an insignificant , it is very likely that is in neither nor . We can also see that genetically distinct groups have distributions with little overlap (Figure 1), and so it may be worth investigating the utility of Equations 1,2 for ancestry inference.

On this note, let us once more consider the quantity which Equation 1 measures, namely the distance of from relative to the distance of from . Referring to Figure 1A and 1C, we can see that samples which are more like those in sample have a distribution that lies to the right of samples which are more similar to , as expected; that is, in Figure 1A and 1C, the distribution of null (not in ) CGEMS cases (dashed red line) is shifted to the right with respect to the distribution of null CGEMS controls, as might be expected from Equation 1, i.e., the CGEMS case Ys are closer to CGEMS case Gs than are the CGEMS control Ys. Although this difference is not statistically significant, one could imagine that it may be possible to select SNPs for which the shift is significant, i.e., a selection of SNPs for which unknown cases are statistically more likely to be closer (via Equation 1) to the cases in and unknown controls are statistically more likely to be closer to the controls in . In this case, a subset of SNPs known to be associated with disease may potentially be used with Equations 1, 2 to predict the case status of new individuals; conversely, finding a subset of SNPs which produce significant separations of the test samples may be indicative of a group of SNPs which play a role in disease. Because this type of application would use fewer SNPs and would involve the comparison of two distributions of (cases vs. controls ), it may be possible to circumvent some of the problems stemming from the unknown width and location of the null distribution described above; still, much work is needed to investigate this possible application. If successful, the metric proposed in [1], while failing to function as a tool to positively identify the presence of a specific individual's DNA in a finite genetic sample, may if refined be a useful tool in the analysis of GWAS data.