PLoS Genet plos plosgen PLOS Genetics 1553-7390 1553-7404 Public Library of Science San Francisco, CA USA PGENETICS-D-19-01506 10.1371/journal.pgen.1009241 Research Article Biology and life sciencesGeneticsHeredityInbreeding Physical sciencesMathematicsStatisticsStatistical models Biology and life sciencesComputational biologyGenome analysisGenome-wide association studies Biology and life sciencesGeneticsGenomicsGenome analysisGenome-wide association studies Biology and life sciencesGeneticsHuman geneticsGenome-wide association studies Research and analysis methodsSimulation and modeling Biology and life sciencesGeneticsHeredityGenetic mappingVariant genotypes Biology and life sciencesGeneticsHeredity Biology and life sciencesEvolutionary biologyPopulation genetics Biology and life sciencesGeneticsPopulation genetics Biology and life sciencesPopulation biologyPopulation genetics Biology and life sciencesGeneticsGenetic loci Estimating FST and kinship for arbitrary population structures Estimating FST and kinship for arbitrary population structures https://orcid.org/0000-0003-4928-3403 Ochoa Alejandro Conceptualization Formal analysis Funding acquisition Investigation Methodology Software Writing – original draft Writing – review & editing 1 2 https://orcid.org/0000-0001-5992-402X Storey John D. Conceptualization Formal analysis Funding acquisition Investigation Methodology Writing – original draft Writing – review & editing 3 * Duke Center for Statistical Genetics and Genomics, Duke University, Durham, North Carolina, United States of America Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina, United States of America Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, New Jersey, United States of America Feldman Marcus W. Editor Stanford University, UNITED STATES

The authors have declared that no competing interests exist.

 * E-mail: jstorey@princeton.edu 1 2021 19 1 2021 17 1 e1009241 11 9 2019 2 11 2020 2021 Ochoa, Storey This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

FST and kinship are key parameters often estimated in modern population genetics studies in order to quantitatively characterize structure and relatedness. Kinship matrices have also become a fundamental quantity used in genome-wide association studies and heritability estimation. The most frequently-used estimators of FST and kinship are method-of-moments estimators whose accuracies depend strongly on the existence of simple underlying forms of structure, such as the independent subpopulations model of non-overlapping, independently evolving subpopulations. However, modern data sets have revealed that these simple models of structure likely do not hold in many populations, including humans. In this work, we analyze the behavior of these estimators in the presence of arbitrarily-complex population structures, which results in an improved estimation framework specifically designed for arbitrary population structures. After generalizing the definition of FST to arbitrary population structures and establishing a framework for assessing bias and consistency of genome-wide estimators, we calculate the accuracy of existing FST and kinship estimators under arbitrary population structures, characterizing biases and estimation challenges unobserved under their originally-assumed models of structure. We then present our new approach, which consistently estimates kinship and FST when the minimum kinship value in the dataset is estimated consistently. We illustrate our results using simulated genotypes from an admixture model, constructing a one-dimensional geographic scenario that departs nontrivially from the independent subpopulations model. Our simulations reveal the potential for severe biases in estimates of existing approaches that are overcome by our new framework. This work may significantly improve future analyses that rely on accurate kinship and FST estimates.

 Author summary

Kinship coefficients and FST, which measure relatedness and population structure, respectively, are important quantities needed to accurately perform various analyses on genetic data, including genome-wide association studies and heritability estimation. However, existing estimators require restrictive assumptions of independence that are not met by real human and other datasets. In this work we find that existing estimators can be severely biased under reasonable scenarios, first by theoretically determining their properties, and then using an admixture simulation to illustrate our findings. In particular, we find that existing FST estimators are downwardly biased, and that existing kinship matrix estimators have related biases that are on average downward and of similar magnitude but vary for every pair of individuals. These insights led us to a new estimation framework for kinship and FST that is practically unbiased for any population structure, as demonstrated by theory and simulations. Our new approaches—available as open-source R packages—are easy to use and are more widely applicable than existing approaches, and they are likely to improve downstream analyses that require accurate kinship and FST estimates.

 http://dx.doi.org/10.13039/100000051 National Human Genome Research Institute R01 HG006448 https://orcid.org/0000-0001-5992-402X Storey John D. This research was supported in part by National Institutes of Health, National Human Genome Research Institute grant R01 HG006448 (JDS). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. PLOS Publication Stage vor-update-to-uncorrected-proof Publication Update 2021-01-29 Data Availability The data and computer code for these manuscripts can be found at https://github.com/StoreyLab/human-differentiation-manuscript. Introduction

In population genetics studies, one is often interested in characterizing structure, genetic differentiation, and relatedness among individuals. Two quantities often considered in this context are FST and kinship. FST is a parameter that measures structure in a subdivided population, satisfying FST = 0 for an unstructured population and FST = 1 if every locus has become fixed for some allele in each subpopulation. More generally, FST is the probability that alleles drawn randomly from a subpopulation are “identical by descent” (IBD) relative to an ancestral population [1, 2]. The kinship coefficient is a measure of relatedness between individuals defined in terms of IBD probabilities, and it is closely related to FST, since the mean kinship of the parents in a subpopulation is the FST of the following generation [1].

This work focuses on the estimation of FST and kinship from biallelic single-nucleotide polymorphism (SNP) data. Existing estimators can be classified into parametric estimators (methods that require a likelihood function) and non-parametric estimators (such as the method-of-moments estimators we focus on, which only require low-order moment equations). There are many likelihood approaches that estimate FST and kinship, but these are limited by assuming independent subpopulations or Normal approximations for FST [3–11] or totally outbred individuals for kinship [12, 13]. Additionally, more complete likelihood models such as that of Jacquard [14] are underdetermined for biallelic loci [15]. Non-parametric approaches such as those based on the method of moments are considerably more flexible and computationally tractable [16], so they are the natural choice to study arbitrary population structures.

The most frequently-used FST estimators are derived and justified under the “independent subpopulations model,” in which non-overlapping subpopulations evolved independently by splitting all at the same time from a common ancestral population. The Weir-Cockerham (WC) FST estimator assumes subpopulations of differing sample sizes and equal per-subpopulation FST relative to the common ancestral population [17]. The Weir-Hill FST estimator generalized WC for subpopulations with different FST values, and first considered arbitrary coancestry between subpopulations, resulting in estimates of a linearly-transformed FST, namely (FST-θ˜)/(1-θ˜) (where θ˜ is the unknown mean coancestry value between subpopulations) [4, 18, 19]. Weir-Hill has further evolved into the Weir-Goudet approach, incorporating relatedness for subpopulations and individuals based on allele matching, also estimating a linearly-transformed FST [20–22]. Note that the Weir-Hill and Weir-Goudet approaches intended to estimate such linearly-transformed quantities, which may be negative, and they did not aim to estimate IBD probabilities [4, 18–22]; in contrast, our goal is to estimate IBD probabilities, which must be non-negative and valid probabilities. The “Hudson” FST estimator [23] assumes two subpopulations with different FST values. All of the previous FST estimators are ratio estimators derived using the method of moments to have unbiased numerators and denominators, which gives approximately unbiased ratio estimates when their assumptions are met [4, 17, 23]. We also evaluate BayeScan [10], which estimates population-specific FST values using a Bayesian model and the Dirichlet-Multinomial likelihood function—thus representing non-method-of-moments approaches—but which like other existing FST estimators also assumes that subpopulations are non-overlapping and evolve independently. These FST estimators are important contributions, used widely in the field.

Kinship coefficients are now commonly calculated in population genetics studies to capture structure and relatedness. Kinship is utilized in principal components analyses and linear-mixed effects models to correct for structure in Genome-Wide Association Studies (GWAS) [16, 24–30] and to estimate genome-wide heritability [31, 32]. Often absent in previous models is a clear identification and role of the ancestral population T that sets the scale of the kinship estimates used. Omission of T makes sense when kinship is estimated on an unstructured population (where only a few individual pairs are closely related; there T is the current population). Our more complete notation brings T to the fore and highlights its key role in kinship estimation and its applications. The most commonly-used kinship estimator [16, 27, 30–36] is also a method-of-moments estimator whose operating characteristics are largely unknown in the presence of structure. We show here that this popular estimator is accurate only when the average kinship is zero, which implies that the population must be unstructured.

The goal of our work is to consistently estimate IBD probabilities, namely kinship coefficients and FST, for which there are currently no consistent estimators under general relatedness. Estimation of these as probabilities, as opposed to linearly-transformed quantities that may be negative, is important since the probabilistic definition of these parameters was required to derive their fundamental connections to many applications in genetics, including allele fixation [1, 2, 37], DNA forensics [3], and heritability [38, 39]. Although IBD probabilities are not absolute, but rather depend on the choice of ancestral population [40], their values become fixed upon agreeing to estimate them in terms of the Most Recent Common Ancestor (MRCA) population, which has long been the choice for models of FST [17, 23, 41] and kinship estimation from pedigrees [42, 43] or markers [12, 13].

Recent genome-wide studies have revealed that humans and other natural populations are structured in a complex manner that break the assumptions of the above estimators. Such complex population structures has been observed in several large human studies, such as the Human Genome Diversity Project [44, 45], the 1000 Genomes Project [46], Human Origins [47–49], and other contemporary [50–54] and archaic populations [55, 56]. We have also demonstrated that the global human population has a complex kinship matrix and no independent subpopulations [57–59]. Therefore, there is a need for innovative approaches designed for complex population structures. To this end, we reveal the operating characteristics of these frequently-used FST and kinship estimators in the presence of arbitrary forms of structure, which leads to a new estimation strategy for FST and kinship.

Here, we study existing FST and kinship method-of-moments estimators in models that allow for arbitrary population structures (see Fig 1 for an overview of the results). First, in section The generalized FST for arbitrary population structures we present the generalized definition of FST for arbitrary population structures [57]. In section The kinship and coancestry models we review the kinship model for genotype covariance [1, 14] and the coancestry model for individual-specific allele frequencies [57, 60, 61]. In section Assessing the accuracy of genome-wide ratio estimators we obtain new strong convergence results for a family of ratio estimators that includes the most common FST and kinship estimators. Next, we calculate the convergence values of these FST (section FST estimation based on the independent subpopulations model) and kinship (section Characterizing a kinship estimator and its relationship to FST) estimators under arbitrary population structures, where we find biases that are not present under their original assumptions about structure (panels “Indep. Subpop. FST Estimator” and “Existing Kinship Estimator” in Fig 1). We characterize the limit of the standard kinship estimator, identifying complex biases or distortions, in agreement with recent work [21, 62].

10.1371/journal.pgen.1009241.g001 Accuracy of <italic>F</italic><sub>ST</sub> and kinship estimators: Overview of models and results.

Our analysis is based on the generalized FST definition (section The generalized FST for arbitrary population structures) and two parallel models: the “Coancestry Model” for individual-specific allele frequencies (πij), and the “Kinship Model” for genotypes (xij). The “Coancestry in Terms of Kinship” panel connects kinship (φjkT, fjT) and coancestry (θjkT) parameters (section The kinship and coancestry models). We use these models to study the accuracy of FST and kinship method-of-moment estimators under arbitrary population structures. The “Indep. Subpop. FST Estimator” panel shows the bias resulting from the misapplication of FST estimators for independent subpopulations (F^STindep) to arbitrary structures (section FST estimation based on the independent subpopulations model), as calculated under the coancestry model. The “Existing Kinship Estimator” panel shows the bias in the standard kinship model estimator (φ^jkT,std) and its resulting plug-in FST estimator (F^STstd; section Characterizing a kinship estimator and its relationship to FST), as calculated under the kinship model. The “New Kinship Estimator” panel presents a new statistic Ajk that estimates kinship with a uniform bias, which together with a consistent estimator of its minimum value (A^min) results in our new kinship (φ^jkT,new) and FST (F^STnew) estimators, which are consistent under arbitrary population structure (section A new approach for kinship and FST estimation).

In section A new approach for kinship and FST estimation we introduce a new approach for kinship and FST estimation for arbitrary population structures, and demonstrate the improved performance using a simple implementation of these estimators (panel “New Kinship Estimator” in Fig 1). There are two key innovations. First, based on the method of moments, we derive a statistic that estimates kinship coefficients up to a shared unknown scaling factor. Second, we propose a new condition, the identification of unrelated individual pairs in the data, which yields the value of the unknown scaling factor and enables the consistent estimation of kinship matrices and FST. We present a simple implementation of this second estimator, based on taking the minimum average statistic value between subpopulations, which in section Simulations evaluating FST and kinship estimators is shown to perform well under some misspecification, namely in an admixture scenario that does not actually have subpopulations [63–65]. Elsewhere, we analyze the Human Origins and 1000 Genomes Project datasets with our novel kinship and FST estimation approach, where we demonstrate its coherence with the African Origins model, and illustrate the shortcomings of previous approaches in these complex data [59]. In summary, we identify a new approach for unbiased estimation of FST and kinship, and we provide new estimators that are nearly unbiased.

 Results The generalized <italic>F</italic><sub>ST</sub> for arbitrary population structures

The existing FST definition requires individuals to belong to discrete, non-overlapping subpopulations, so it must be generalized in order to apply to arbitrary population structures (such as the admixture model with individual-specific ancestry proportions considered in our simulations). Our generalized FST can be understood as a two-step strategy: (1) we define FST on a per-individual basis, and (2) we define FST for a group of individuals as a weighted average of the per-individual FST values [57].

The inbreeding coefficient fjT of an individual j relative to an ancestral population T is defined as the probability that the two alleles at a random locus are identical by descent (IBD) [37]. Note that the ancestral population T determines what is IBD: only relationships since T count toward IBD. This total inbreeding coefficient (fjT) is the individual analog of Wright’s total inbreeding coefficient FIT, the latter of which is the mean fjT over a group of individuals [2]. Wright partitioned total inbreeding (FIT) into local (FIS) and structural (FST) coefficients defined by a subpopulation S that contains all individuals in question and evolved from the ancestral population T, so that FIS is the inbreeding of individuals relative to S (as opposed to T) and FST is inbreeding of the subpopulation S relative to T, and these coefficients satisfy (1 − FIT) = (1 − FIS)(1 − FST) [2]. In our generalized definitions for one individual j, we restrict the subpopulation of interest (S) to be Lj, called the local subpopulation of j, which is the most recent subpopulation from which j drew its alleles. In this case, fjLj is the local inbreeding coefficient of j (always relative to its local subpopulation Lj), and fLjT is the structural inbreeding coefficient of j (equal to the inbreeding of the subpopulation Lj relative to T), and being a special case of Wright’s equation, they also satisfy [57] (1-fjT)=(1-fjLj)(1-fLjT).

Now we discuss estimating the three quantities we just introduced. First, the total inbreeding coefficient (fjT) should be estimated from the variance of genotypes, using the practically unbiased approach we introduce in this work. Second, note that the local inbreeding coefficient (fjLj) corresponds to (non-population) family relatedness, so it can be taken to be the inbreeding calculated from a pedigree if it is available [42]. Note that estimation of the various inbreeding coefficients from pedigrees was the only approach available to Wright when he studied cattle and defined inbreeding and FST [2, 37]. Alternatively, in the absence of pedigrees, local inbreeding can be estimated from inferred self-IBD blocks or unusually-large runs of homozygosity [66–68]. Lastly, since the structural inbreeding coefficient (fLjT) is given by the previous two quantities (solving from Eq (1)) by fLjT=fjT-fjLj1-fjLj, then we propose estimating fLjT using this equation, from the above estimates of fjT and fjLj.

As a toy example, suppose we estimate a total inbreeding coefficient of fjT=0.15 for a given individual whose parents are first cousins, then the pedigree expectation for its local inbreeding is fjLj=116=0.0625, and the structural inbreeding (i.e. the FST of this individual) using Eq (2) is fLjT≈0.093. However, if in the same example (fjT=0.15) the individual instead had parents who were second cousins, then fjLj=164≈0.0156, then the structural estimate becomes fLjT≈0.137, which is much closer to the total inbreeding value. Thus, when total inbreeding estimates are much larger than local inbreeding estimates, correcting for the latter via Eq (2) may not change the numerical estimate of structural inbreeding by a meaningful amount. Conversely, as the local inbreeding coefficient is reduced exponentially with the degree of relatedness of the parents (fjLj=14n+1 for n-th cousins), and as local inbreeding is required to be recent (to exclude population-level inbreeding), then sufficiently-accurate estimates of structural inbreeding can be obtained by estimating non-zero local inbreeding only for individuals with the most related parent pairs (above a certain degree of relatedness).

We define the generalized FST across n individuals as the weighted average of the per-individual structural inbreeding coefficients (i.e., individual FST values) [57], FST=∑j=1nwjfLjT, where wj is the weight of individual j and the weights are required to sum to one and be non-negative. The above is a straightforward generalization of Wright’s FST: if every individual j has Lj = S as its local subpopulation, then Eq (3) becomes FST=∑j=1nwjfST=fST, where fST is the inbreeding coefficient of subpopulation S relative to T, so it has the same meaning as Wright’s FST (the exact weights here do not matter as long as ∑j=1nwj=1, as required). Moreover, if each individual j belongs to one of K subpopulations Su (u ∈ {1, …, K}) and if subpopulations are weighted equally (∑j∈Suwj=1K for every Su), then Eq (3) becomes FST=1K∑u=1KfSuT, so it equals the (unweighted) average subpopulation-specific FST (i.e., fSuT), which is the FST definition for multiple subpopulations prevalent in modern work [4, 21, 23]. The last case illustrates the need for weights, which above downweights individuals that belong to subpopulations with greater numbers of observations. In general, weights allow adjustment for skewed or unbalanced samples. However, in complicated scenarios without subpopulations and no obvious sampling biases, for simplicity we recommend using uniform weights (wj=1n) for the target generalized FST.

In terms of total and local inbreeding coefficients (using Eq (2)), the generalized FST equals FST=∑j=1nwjfjT-fjLj1-fjLj, which immediately suggests the estimation strategy when estimates of the total and local inbreeding coefficients are available. For simplicity, in the remainder of this work we shall consider only locally-outbred individuals (fjLj=0 for all j), for which the generalized FST simply equals the weighted mean total inbreeding coefficient: FST=∑j=1nwjfjT. This greatly simplifies our discussion of bias for all of the FST estimators we analyzed; determining the statistical properties of local inbreeding estimators is beyond the scope of this work. Moreover, the assumption of locally-outbred individuals is satisfied in all of the simulations presented in this work.

 The kinship and coancestry models

The generalized FST above is given solely in terms of inbreeding coefficients. In order to establish our results and framework, it is necessary to consider kinship coefficients as well. The kinship coefficient is the extension of the inbreeding coefficient for a pair of individuals: the kinship coefficient φjkT of two individuals j and k relative to an ancestral population T is the probability that two alleles, chosen at random from each individual at a random locus, are IBD [1]. Note that the self-kinship coefficient is related to the inbreeding coefficient by φjjT=12(1+fjT) [16].

Kinship coefficients determine the covariance structure of genotypes, which is the key to estimating kinship and FST from genotype data. We shall concentrate on biallelic variants, which include single-nucleotide polymorphisms, and are the dominant data from genotyping microarrays and whole-genome sequencing studies. We shall also restrict our attention to diploid organisms in this present work. Genotypes are encoded into variables xij for each locus i and individual j that count the number of alleles (dosage) of a given reference type, so for diploid organisms xij = 2 is homozygous for the reference allele, xij = 0 is homozygous for the alternative allele, and xij = 1 is heterozygous. Based on the definition of the IBD probabilities, the kinship model determines the mean and covariance structure of the genotype random variables at neutral loci [1, 2, 14, 16, 37]: E[xij|T]=2piT,Cov(xij,xik|T)=4piT(1−piT)φjkT, where piT is the allele frequency at locus i in the ancestral population T and 0<piT<1.

The coancestry model resembles the kinship model, but it is formulated in terms of allele frequencies, which simplifies our analysis of FST estimators for subpopulations as well as yielding kinship coefficients under the admixture model we simulate from in this work. Let πij be the individual-specific allele frequency (IAF) at locus i for individual j, which is a real number between zero and one [60, 61]. Our coancestry model assumes that [57] E[πij|T]=piT,Cov(πij,πik|T)=piT(1-piT)θjkT, where θjkT is the coancestry coefficient between individuals j and k relative to the ancestral population T. This model is inspired by coancestry models for subpopulations common in the FST literature [4, 5, 21, 23], and exactly equals those models when subpopulation sizes go to infinity, in which case j and k index subpopulations rather than individuals, and πij is interpreted as the true allele frequency at locus i for subpopulation j.

The coancestry model connects to the kinship model under the additional assumption that the alleles of an individual j are drawn independently from its IAF, xij|πij∼Binomial(2,πij). In this case, marginalizing the intermediate IAF random variables (πij) and matching the resulting genotype moments results in the following equivalence [57]: θjkT={fjTifj=k,φjkTifj≠k. The coancestry coefficient equals the kinship coefficient between two different individuals, but the self-coancestry coefficient equals the inbreeding coefficient (rather than the self-kinship coefficient). However, since in the coancestry model alleles are drawn independently conditional on the IAF in Eq (7), then the only structure present is the population structure, so these coancestry models cannot generate family structures, unlike the more general kinship model that also encompasses pedigrees. Therefore, despite Eq (8), the kinship and coancestry are not equivalent models except under the more restrictive assumptions of the coancestry model. Thus, individuals drawn from this model are always locally-outbred, so θjjT=fLjT also equals the structural inbreeding coefficient, and the generalized FST under the coancestry model is therefore FST=∑j=1nwjθjjT, which also generalizes previous definitions of FST under coancestry for subpopulations [4, 5, 21, 23]. The kinship and coancestry models, and their connection, is included in the overview Fig 1.

 Assessing the accuracy of genome-wide ratio estimators

In this section we change gears to focus on theoretical convergence properties of two broad estimator families. The resulting theory will be applied repeatedly to various FST and kinship estimators of interest in later sections.

Many FST and kinship coefficient method-of-moments estimators are ratio estimators, a general class of estimators that tends to be biased and to have no closed-form expectation [69]. In the FST literature, the expectation of a ratio is frequently approximated with a ratio of expectations [4, 17, 23]. Specifically, ratio estimators are often called “unbiased” if the ratio of expectations is unbiased, even though the ratio estimator itself may be biased [69]. Here we characterize the behavior of two ratio estimator families calculated from genome-wide data, known as “ratio-of-means” and “mean-of-ratios” estimators [23], detailing conditions where the previous approximation is justified and providing additional criteria to assess the accuracy of such estimators.

Ratio estimators

The general problem of forming ratio estimators involves random variables ai and bi calculated from genotypes at each locus i, such that E[ai] = Aci and E[bi] = Bci and the goal is to estimate AB. A and B are constants shared across loci (given by FST or φjkT), while ci depends on the ancestral allele frequency piT and varies per locus. The problem is that the single-locus estimator aibi is biased, since E[aibi]≠E[ai]E[bi]=AB, which applies to ratio estimators in general [69]. Below we study two estimator families that combine large numbers of loci to better estimate AB.

 Convergence

The solution we recommend is the “ratio-of-means” estimator A^mB^m, where A^m=1m∑i=1mai, and B^m=1m∑i=1mbi, which is common for FST estimators [4, 17, 19, 23, 70]. Note that E[A^m]=Ac¯m and E[B^m]=Bc¯m, where c¯m=1m∑i=1mci. We will assume bounded terms (|ai|, |bi| ≤ C for some finite C), a convergent c¯m→c, and Bc ≠ 0, which are satisfied by common estimators. Given independent loci, we prove almost sure convergence to the desired quantity (S1 Text), A^mB^m=1m∑i=1mai1m∑i=1mbi→m→∞a.s.AB, a strong result that implies E[A^mB^m]→AB, justifying previous work [4, 17, 23]. Moreover, the error between these expectations scales with 1m (S1 Text), just as for standard ratio estimators [69]. Although real loci are not independent due to genetic linkage, their dependence is very localized, so this estimator will perform well if the effective number of independent loci is large.

In order to test if a given ratio-of-means estimator converges to its ratio of expectations as in Eq (10), the following three conditions can be tested. (i) The expected values of each term ai, bi must be calculated and shown to be of the form E[ai] = Aci and E[bi] = Bci for some A and B shared by all loci i and some ci that may vary per locus i but must be shared by both E[ai], E[bi]. In the estimators we study, A and B are functions of IBD probabilities such as φjkT and FST, while ci is a function of piT only. (ii) The mean ci must converge to a non-zero value for infinite loci. (iii) Both |ai|, |bi| ≤ C must be bounded for all i by some finite C (the estimators we study usually have C = 1 or C = 4). If these conditions are satisfied, then Eq (10) holds for independent loci and the A and B found in the first step. See the next section for an example application of this procedure to an FST estimator.

Another approach is the “mean-of-ratios” estimator 1m∑i=1maibi, used often to estimate kinship coefficients [16, 27, 30–35] and FST [46]. If each aibi is biased, their average across loci will also be biased, even as m → ∞. However, if E[aibi]→AB for all loci i = 1, …, m as the number of individuals n → ∞, and Var(aibi) is bounded, then 1m∑i=1maibi→n,m→∞a.s.AB. Therefore, mean-of-ratios estimators must satisfy more restrictive conditions than ratio-of-means estimators, as well as large n (in addition to the large m needed by both estimators), to estimate AB well. We do not provide a procedure to test whether a given mean-of-ratios estimator converges as shown above.

 <italic>F</italic><sub>ST</sub> estimation based on the independent subpopulations model

Now that we have detailed how ratio estimators may be evaluated for their accuracy, we turn to existing estimators and assess their accuracy under arbitrary population structures. We study the FST estimators Weir-Cockerham (WC) [17], Weir-Hill [4], “Hudson” [23], and Weir-Goudet (equals HudsonK below for biallelic loci; S1 Text) [21]. The panel “Indep. Subpop. FST Estimator” in Fig 1 provides an overview of our results, which we detail in this section.

The <italic>F</italic><sub>ST</sub> estimator for independent subpopulations and infinite subpopulation sample sizes

The WC, Weir-Hill, and Hudson method-of-moments estimators have small sample size corrections that remarkably make them consistent (as the number of independent loci m goes to infinity) for finite numbers of individuals. However, these small sample corrections also make the estimators unnecessarily cumbersome for our purposes (see Methods, section Previous FST estimators for the independent subpopulations model for complete formulas). In order to illustrate clearly how these estimators behave, both under the independent subpopulations model and for arbitrary structure, here we construct simplified versions that assume infinite sample sizes per subpopulation (Methods, section Previous FST estimators for the independent subpopulations model). This simplification corresponds to eliminating statistical sampling, leaving only genetic sampling to analyze [71]. Note that our simplified estimator nevertheless illustrates the general behavior of the WC, Weir-Hill, and Hudson estimators under arbitrary structure, and the results are equivalent to those we would obtain under finite sample sizes of individuals. While the Hudson FST estimator compares two subpopulations [23], based on that work we derive a generalized “HudsonK” estimator for more than two subpopulations in Methods, section Generalized HudsonK FST estimator. Note that HudsonK, first derived in [58], also equals the Weir-Goudet FST estimator for subpopulations [21] when loci are biallelic, which was derived independently using allele matching (S1 Text).

Under infinite subpopulation sample sizes, the allele frequencies at each locus and every subpopulation are known. Let j ∈ {1, …, n} index subpopulations rather than individuals and πij be the true allele frequency in subpopulation j at locus i. Note that πij are not estimated allele frequencies, but rather true subpopulation allele frequencies; this abstraction does not result in a practical estimation approach, but it greatly simplifies understanding of bias for subpopulations in a setting where there there is no statistical sampling. Although in this analysis of FST estimators the πij values are applied to subpopulations, for coherence with our previous work we shall call them “individual-specific allele frequencies” (IAF) [60, 61]. Whether for individuals or subpopulations, the key assumption is that IAFs satisfy the coancestry model of Eq (6). In this special case of infinite subpopulation sample sizes, all of WC, Weir-Hill, and HudsonK simplify to the following FST estimator for independent subpopulations (“indep”; derived in Methods, section Previous FST estimators for the independent subpopulations model): p^iT=1n∑j=1nπij, σ^i2=1n-1∑j=1n(πij-p^iT)2, F^STindep=∑i=1mσ^i2∑i=1mp^iT(1−p^iT)+1nσ^i2. The goal is to estimate FST=1n∑j=1nθjjT, which is the special case of Eq (9) that weighs every subpopulation j equally (wj=1n∀j).

 <italic>F</italic><sub>ST</sub> estimation under the independent subpopulations model

Under the independent subpopulations model θjkT=0 for j ≠ k, where T is the most recent common ancestor (MRCA) population of the set of subpopulations. Note that the estimator in Eq (13) can be derived directly from Eq (6) and these assumptions using the method of moments (ignoring the existence of previous FST estimators; S1 Text). The expectations of the two recurrent terms in Eq (13) are E[1m∑i=1mσ^i2|T]=p(1-p)¯TFST,E[1m∑i=1mp^iT(1-p^iT)|T]=p(1-p)¯T(1-FSTn),wherep(1-p)¯T=1m∑i=1mpiT(1-piT). Eliminating p(1-p)¯T and solving for FST in this system of equations recovers the estimator in Eq (13).

Before applying the convergence result in Eq (10), we test that the three conditions listed in section Assessing the accuracy of genome-wide ratio estimators are met. Condition (i): The locus i terms are ai=σ^i2 and bi=p^iT(1-p^iT)+1nσ^i2, which satisfy E[ai] = Aci and E[bi] = Bci with A = FST, B = 1, and ci=piT(1-piT). Condition (ii): c¯m→c=E[piT(1-piT)|T]≠0 over the piT distribution across loci. Condition (iii): Since 0≤πij,p^iT≤1, then 0≤σ^i2≤1 and 0≤p^iT(1-p^iT)≤14, and since n ≥ 2, C = 1 bounds both |ai| and |bi|. Therefore, for independent loci, F^STindep→m→∞a.s.FST.

 <italic>F</italic><sub>ST</sub> estimation under arbitrary coancestry

Now we consider applying the independent subpopulations FST estimator to dependent subpopulations. The key difference is that now θjkT≠0 for every (j, k) will be assumed in our coancestry model in Eq (6), and now T may be either the MRCA population of all subpopulations or a more ancestral population. In this general setting, (j, k) may index either subpopulations or individuals. The two terms of F^STindep now satisfy E[1m∑i=1mσ^i2|T]=p(1-p)¯T(FST-θ¯T)nn-1,E[1m∑i=1mp^iT(1-p^iT)|T]=p(1-p)¯T(1-θ¯T), where θ¯T=1n2∑j=1n∑k=1nθjkT is the mean coancestry with uniform weights. There are two equations but three unknowns: FST, θ¯T, and p(1-p)¯T. The independent subpopulations model satisfies θ¯T=1nFST, which allows for the consistent estimation of FST. Therefore, the new unknown θ¯T precludes consistent FST estimation without additional assumptions. As shown later, our additional assumption is that we can identify unrelated individuals in the data, which determines all unknowns. We defer our complete solution to this problem until kinship and its estimation challenges have been presented.

The FST estimator for independent subpopulations converges more generally to F^STindep→m→∞a.s.FST-θ˜T1-θ˜T, (the conclusion of panel “Indep. Subpop. FST Estimator” in Fig 1), where θ˜T=1n-1(nθ¯T-FST)=1n(n-1)∑j≠kθjkT is the average of all between-subpopulation coancestry coefficients, in agreement with related calculations regarding the WC and Weir-Hill estimators [4, 21]. Therefore, under arbitrary structure the independent subpopulations estimator’s bias is due to the coancestry between subpopulations. While the limit in Eq (14) appears to vary depending on the choice of T, it is in fact a constant with respect to T (proof in S1 Text).

Since 1nFST≤θ¯T≤FST (S1 Text), this estimator has a downward bias in the general setting: it is asymptotically unbiased (F^STindep→m→∞a.s.FST) only when θ¯T=1nFST, while bias is maximal when θ¯T=FST, where F^STindep→m→∞a.s.0. For example, if minθjkT=0 so the MRCA population T is fixed, but n is large and θjkT≈FST for most pairs of subpopulations, then θ¯T≈FST as well, and F^STindep≈0. Therefore, the magnitude of the bias of F^STindep is unknown if θ¯T is unknown, and small F^STindep estimates may arise even if FST is very large.

 Coancestry estimation as a method of moments

Since the generalized FST is given by coancestry coefficients θjjT in Eq (9), a new FST estimator could be derived from estimates of θjjT. Here we attempt to define a method-of-moments estimator for θjkT, and find an underdetermined estimation problem, just as for FST. This is consistent with IBD parameters in general requiring a reference population to be determined [40], whereas in this subsection this reference population is unspecified.

Given IAFs and the coancestry model of Eq (6), the first and second moments that average across loci are E[1m∑i=1mπij|T]=p¯T, E[1m∑i=1mπijπik|T]=p2¯T+p(1-p)¯TθjkT, where p¯T=1m∑i=1mpiT, p2¯T=1m∑i=1m(piT)2, and p(1-p)¯T is as before.

Suppose first that only θjjT are of interest. There are n estimators given by Eq (16) with j = k, each corresponding to an unknown θjjT. However, all these estimators share two nuisance parameters: p¯T and p2¯T. While p¯T can be estimated from Eq (15), there are no more equations left to estimate p2¯T, so this system is underdetermined. The estimation problem remains underdetermined if all n(n+1)2 estimators in Eq (16) are considered rather than only the j = k cases. Therefore, we cannot estimate coancestry coefficients consistently using only the first two moments without additional assumptions.

 Characterizing a kinship estimator and its relationship to <italic>F</italic><sub>ST</sub>

Given the biases we see for F^STindep under arbitrary structures in the previous section, we now turn to the generalized definition of FST and pursue an estimate of it. Recall that our generalized FST in Eq (3) is defined in terms of inbreeding coefficients, which are a special case of the kinship coefficient. Kinship coefficients also determine the bias of F^STindep in Eq (14) (since coancestry and kinship coefficients are closely related: see panel “Coancestry in Terms of Kinship” in Fig 1). Therefore, we will consider estimates of kinship and inbreeding in this section. Estimating kinship is also important for GWAS approaches that control for population structure [16, 24–35, 72, 73].

In this section, we focus on a standard kinship method-of-moments estimator and calculate its limit for the first time (panel “Existing Kinship Estimator” in Fig 1). We study estimators that use genotypes or IAFs, and construct FST estimators from their kinship estimates. We find biases comparable to those of F^STindep in the previous section, and define unbiased FST estimators that require knowing the mean kinship or coancestry, or its proportion relative to FST. The results of this section directly motivate and help construct our new kinship and FST estimation approach in the following section.

Characterization of the standard kinship estimator

Here we analyze a standard kinship estimator that is frequently used [16, 27, 30–36]. We generalize this estimator to use weights in estimating the ancestral allele frequencies, and we write it as a ratio-of-means estimator due to the favorable theoretical properties of this format as detailed in the earlier section Assessing the accuracy of genome-wide ratio estimators: p^iT=12∑j=1nwjxij, φ^jkT,std=∑i=1m(xij−2p^iT)(xik−2p^iT)4∑i=1mp^iT(1−p^iT). The estimator in Eq (18) resembles the sample covariance estimator applied to genotypes, but centers by locus i rather than by individuals j and k, and normalizes using estimates of 4piT(1-piT). We derive Eq (18) directly using the method of moments in S1 Text. The weights in Eq (17) must satisfy wj > 0 and ∑j=1nwj=1, so that 0≤p^iT≤1 and E[p^iT|T]=piT.

Utilizing the kinship model for genotypes from Eq (5), we find that Eq (18) converges to φ^jkT,std→m→∞a.s.φjkT-φ¯jT-φ¯kT+φ¯T1-φ¯T, where φ¯jT=∑k′=1nwk′φjk′T and φ¯T=∑j′=1n∑k′=1nwj′wk′φj′k′T, which agrees with related derivations [21, 62]. (This is the conclusion of panel “Existing Kinship Estimator” in Fig 1; see S1 Text for intermediate calculations that lead to Eq (19).) Therefore, the bias of φ^jkT,std varies per pair of individuals j and k. Analogous distortions have been observed for sample covariances of genotypes [74]. The limit of φ^jkT,std in Eq (19) is constant with respect to T (proof in S1 Text). Similarly, inbreeding coefficient estimates derived from Eq (18) converge to f^jT,std=2φ^jjT-1→m→∞a.s.fjT-4φ¯jT+3φ¯T1-φ¯T. The difference between the bias of φ^jkT,std for j ≠ k in Eq (19) and f^jT,std in Eq (20) is visible in the kinship estimates shown toward the end of the results section. The limits of the ratio-of-means versions of two more fjT estimators [32] are, if p^iT uses Eq (17), f^jT,stdII=1−∑i=1mxij(2−xij)2∑i=1mp^iT(1−p^iT)→m→∞a.s.fjT−φ¯T1−φ¯T,f^jT,stdIII=∑i=1mxij2−(1+2p^iT)xij+2(p^iT)22∑i=1mp^iT(1−p^iT)→m→∞a.s.fjT+φ¯T−2φ¯jT1−φ¯T.

The estimators in Eqs (18) and (21) are unbiased when p^iT is replaced by piT [16, 32, 36], and are consistent when p^iT is consistent [60]. Surprisingly, p^iT in Eq (17) is not consistent (it does not converge almost surely to piT) for arbitrary population structures, which is at the root of the bias in Eqs (19) to (21). In particular, although p^iT is unbiased, its variance (S1 Text, and some special cases shown elsewhere, e.g., [19]), Var(p^iT|T)=piT(1-piT)φ¯T, may be asymptotically non-zero as n → ∞, since piT∈(0,1) is fixed and limn→∞φ¯T may take on any value between zero and one for arbitrary population structures. Further, φ¯T→0 as n → ∞ if and only if φjkT=0 for almost all pairs of individuals (j, k). These observations hold for any weights such that wj>0,∑j=1nwj=1. An important consequence is that the plug-in estimate of piT(1-piT) is biased (S1 Text), E[p^iT(1-p^iT)|T]=piT(1-piT)(1-φ¯T), which is present in all estimators we have studied.

 Estimation of coancestry coefficients from IAFs

Here we form a coancestry coefficient estimator analogous to Eq (18) but using IAFs. Assuming the moments in Eq (6), this estimator and its limit are p^iT=∑j=1nwjπij, θ^jkT,std=∑i=1m(πij-p^iT)(πik-p^iT)∑i=1mp^iT(1-p^iT)→m→∞a.s.θjkT-θ¯jT-θ¯kT+θ¯T1-θ¯T, where θ¯jT=∑k=1nwkθjkT and θ¯T=∑j=1n∑k=1nwjwkθjkT are analogous to φ¯jT and φ¯T. Eq (23) generalizes Eq (11) for arbitrary weights. Thus, use of IAFs does not ameliorate the estimation problems we have identified for genotypes. Like Eq (22), p^iT in Eq (23) is not consistent because Var(p^iT|T)=piT(1−piT)θ¯T may not converge to zero for arbitrary population structures, which causes the bias observed in Eq (24).

 <italic>F</italic><sub>ST</sub> estimator based on the standard kinship estimator

Since the generalized FST is defined as a mean inbreeding coefficient in Eq (3), here we study the FST estimator constructed as F^STstd=∑j=1nwjf^jT,std where f^jT,std is the inbreeding estimator derived from the standard kinship estimator. Although f^jT,std is biased, we nevertheless plug it into our definition of FST so that we may study how bias manifests. Note that we do not recommend utilizing this FST estimator in practice, but we find these results informative for identifying how to proceed in deriving new estimators in the following section.

Remarkably, the three fjT estimators in Eqs (20) and (21) give exactly the same plug-in F^STstd if the weights in FST and p^iT in Eq (17) match, namely F^STstd=∑j=1nwjf^jT,std=∑i=1m∑j=1nwj(xij-2p^iT)22∑i=1mp^iT(1-p^iT)-1→m→∞a.s.FST-φ¯T1-φ¯T, where the limit assumes locally-outbred individuals so Eq (4) holds. The analogous FST estimator for IAFs and its limit are F^STstd=∑j=1nwjθ^jjT,std=∑i=1m∑j=1nwj(πij-p^iT)2∑i=1mp^iT(1-p^iT)→m→∞a.s.FST-θ¯T1-θ¯T. The estimators in Eqs (25) and (26) for individuals and their limits resemble those of classical FST estimators for populations of the form σp2p¯(1-p¯) [4, 5]. F^STstd in Eq (26) for subpopulations j with uniform weight and one locus is also GST for two alleles [75]. Compared to F^STindep in Eq (13), F^STstd in Eq (26) admits arbitrary weights and, by forgoing bias correction under the independent subpopulations model, is a simpler target of study.

Like F^STindep in Eq (13), F^STstd in Eqs (25) and (26) are downwardly biased since 0≤φ¯T,θ¯T. F^STstd in Eq (26) may converge arbitrarily close to zero since θ¯T can be arbitrarily close to FST (S1 Text). Moreover, although φ¯T≈θ¯T for large n (see Eq (8) and panel “Coancestry in Terms of Kinship” in Fig 1), in extreme cases φ¯T can exceed FST under the coancestry model (where θ¯T≤φ¯T) and also under extreme local kinship, where F^STstd in Eq (25) converges to a negative value.

 Adjusted consistent oracle <italic>F</italic><sub>ST</sub> estimators and the “bias coefficient”

Here we explore two adjustments to F^STstd from IAFs in Eq (26) that rely on having minimal additional information needed to correct its bias. These “oracle” approaches require information that is not known in practice, but this exercise helps us understand the problem more deeply and finds further connections between the various FST estimators.

If θ¯T is known, the bias in Eq (26) can be reversed, yielding the consistent estimator F^ST′=F^STstd(1-θ¯T)+θ¯T→m→∞a.s.FST. Consistent estimates are also possible if a scaled version of θ¯T is known, namely sT=θ¯TFST=∑j=1n∑k=1nwjwkθjkT∑j=1nwjθjjT, which we call the “bias coefficient” and which has interesting properties. The bias coefficient quantifies the departure from the independent subpopulations model by comparing the mean coancestry (θjkT) to the mean inbreeding coefficient (θjjT), and given FST > 0 satisfies 0 < sT ≤ 1 (S1 Text). The limit in Eq (26) in terms of sT is F^STstd→m→∞a.s.FST1-sT1-sTFST. Treating the limit as equality and solving for FST yields the following consistent estimator: σ^i2=11-sT∑j=1nwj(πij-p^iT)2, F^ST′′=F^STstd1-sT(1-F^STstd)=∑i=1mσ^i2∑i=1mp^iT(1-p^iT)+sTσ^i2→m→∞a.s.FST. Note that σ^i2 and F^STindep from Eqs (12) and (13) are the special case of Eqs (30) and (31) for uniform weights and sT=1n; hence, F^ST′′ generalizes F^STindep.

Lastly, using either Eqs (26) or (29), the relative error of F^STstd converges to 1-F^STstdFST→m→∞a.s.θ¯T(1-FST)FST(1-θ¯T)=sT1-FST1-sTFST, which is approximated by sT if FST ≪ 1, hence the name “bias coefficient”. Note sT varies depending on the choice of T, which is necessary since FST (and hence the relative bias of F^STstd from FST) depends on the choice of T.

 A new approach for kinship and <italic>F</italic><sub>ST</sub> estimation

Here, we propose a new estimation approach for kinship coefficients that has properties favorable for obtaining nearly unbiased estimates (panel “New Kinship Estimator” in Fig 1). These new kinship estimates yield an improved FST estimator. We present the general approach and implement a simple version of one key estimator that results in the complete proof-of-principle estimator that is evaluated in the next section and applied to human data in [59]. We also compare our approach to a related estimator of non-IBD linearly-transformed kinship values [20–22] that was proposed concurrently to ours [58].

General approach

In this subsection we develop our new estimator in two steps. First, we compute a new statistic Ajk that is proportional in the limit of infinite loci to φjkT-1 times a nuisance factor vT. Second, we estimate and remove vT to yield the proposed estimator φ^jkT,new. A^min—an estimator of the limit of the minimum Ajk—yields vT if the least related pair of individuals in the data has φjkT=0, which sets T to the MRCA population of all the individuals in the data. The new kinship estimator immediately results in new inbreeding (f^jT,new) and FST (F^STnew) estimators. This general approach leaves the implementation of A^min open; the simple implementation applied in this work is described in subsection Proof-of-principle kinship estimator using subpopulation labels, but our method can be readily improved by substituting in a better A^min in the future.

Applying the method of moments to Eq (5), we derive the following statistic (S1 Text), whose expectation is proportional to φjkT-1: Ajk=1m∑i=1m(xij-1)(xik-1)-1,E[Ajk|T]=(φjkT-1)vmT,wherevmT=4m∑i=1mpiT(1-piT). Compared to the standard kinship estimator in Eq (19), which has a complex asymptotic bias determined by n parameters (φ¯jT for each j ∈ {1, …, n}), the Ajk statistics estimate kinship with a bias controlled by the sole unknown parameter vmT shared by all pairs of individuals. The key to estimating vmT is to notice that if φjkT=0 then E[Ajk|T]=-vmT. Thus, assuming minj,kφjkT=0, which sets T to the MRCA population, then the minimum Ajk yields the nuisance parameter. However, we recommend using a more stable estimate than the minimum Ajk to unbias all Ajk, such as the estimator presented in the next subsection.

In general, suppose A^min is a consistent estimator of the limit of the minimum E[Ajk|T], or equivalently, A^min→m→∞a.s.-vT, along with the assumption that vmT→m→∞vT for some vT ≠ 0. Our new kinship estimator follows directly from replacing vmT with -A^min and solving for φjkT in Eq (33), which results in a consistent kinship estimator (given the convergence proof of section Assessing the accuracy of genome-wide ratio estimators): φ^jkT,new=1-AjkA^min→m→∞a.s.φjkT. The resulting new inbreeding coefficient estimator is f^jT,new=2φ^jjT,new-1→m→∞a.s.fjT, and the new FST estimator is consistent for locally-outbred individuals (estimates Eq (4)): F^STnew=∑j=1nwjf^jT,new→m→∞a.s.FST. Thus, only the implementation of A^min is left unspecified from this general estimation approach of kinship and FST. The implementation of A^min used in the analyses in this work is given in the next subsection.

 Proof-of-principle kinship estimator using subpopulation labels

To showcase the potential of the new estimators, we implement a simple proof-of-principle version of A^min needed for our new kinship estimator (φ^jkT,new in Eq (34)). This A^min relies on an appropriate partition of the n individuals into K subpopulations (denoted Su for u ∈ {1, …, K}), where the only requirement is that the kinship coefficients between pairs of individuals across the two most unrelated subpopulations is zero, as detailed below. Note that, unlike the the independent subpopulations model of section FST estimation based on the independent subpopulations model, these K subpopulations need not be independent nor unstructured. The desired estimator A^min is the minimum average Ajk over all subpopulation pairs: A^min=minu≠v1|Su||Sv|∑j∈Su∑k∈SvAjk. This A^min consistently estimates the limit of the minimum Ajk if φjkT=0∀j∈Su,∀k∈Sv for the least related pair of subpopulations Su, Sv.

This estimator should work well for individuals truly divided into subpopulations, but may be biased for a poor choice of subpopulations, in particular if the minimum mean φjkT between subpopulations is far greater than zero. For this reason, inspection of the kinship estimates is required and careful construction of appropriate subpopulations may be needed. See our analysis of human data for detailed examples [59]. Future work could focus on a more general A^min that circumvents the need for subpopulations of our proof-of-principle estimator.

 Comparison to the Weir-Goudet kinship estimator for individuals

Here we analyze the Weir-Goudet (WG) kinship estimator for individuals [20–22]. This has connections to our new estimator but differs in having the goal of estimating linearly-transformed kinship values. In our framework, the WG estimator is given by φ^jkT,WG=1-AjkA^avg,whereA^avg=2n(n-1)∑j=2n∑k=1j-1Ajk. Therefore, this estimator differs from our proposal [58] by replacing our A^min with A^avg. Under the kinship model, the expectation of A^avg is E[A^avg|T]=(φ˜T-1)vmT,whereφ˜T=2n(n-1)∑j=2n∑k=1j-1φjkT. Therefore, the limit of this estimator is φ^jkT,WG→m→∞a.s.φjkT−φ˜T1−φ˜T, which agrees with calculations in the original WG work [20–22]. Note that, assuming that kinship coefficients must be non-negative, the above estimator recovers the kinship IBD probabilities if and only if φ˜T=0 which holds if and only if φjkT=0 for every pair of individuals j ≠ k. The resulting WG inbreeding coefficient estimator is f^jkT,WG=2φ^jkT,WG−1→m→∞a.s.fjT−φ˜T1−φ˜T, which estimates linearly-transformed inbreeding values [21]. Therefore, the resulting WG FST estimator (for individuals) also targets a linearly-transformed FST value (under locally-outbred individuals, where FST is given by Eq (4)), namely F^STWG=1n∑j=1nf^jT,WG→m→∞a.s.FST−φ˜T1−φ˜T. The WG authors also briefly consider a variant of their kinship estimator that is normalized using the minimum kinship value as we did, developed concurrently with our approach [58], but was largely dismissed as an unnecessary correction [21, 76]. See S1 Text for a detailed proof that the general estimator framework we propose here (Eqs (33) and (34)) is algebraically equivalent to our original formulation in [58].

Note that the original WG does not estimate FST from individuals as considered above; instead, FST is estimated from coancestry estimates for subpopulations (which equals our HudsonK for biallelic loci, S1 Text) [20–22]. For completeness, we consider both kinds of FST estimates in the evaluations that follow.

 Simulations evaluating <italic>F</italic><sub>ST</sub> and kinship estimators Overview of simulations

We simulate genotypes from two models to illustrate our results when the true population structure parameters are known. Both simulations have clearly-defined IBD probability parameters in terms of the MRCA population. The first simulation satisfies the independent subpopulations model that existing FST estimators assume. The second simulation is from an admixture model with no independent subpopulations and pervasive kinship designed to induce large downward biases in existing kinship and FST estimators (Fig 2). This admixture scenario resembles the population structure we estimated for Hispanics in the 1000 Genomes Project [59]: compare the simulated kinship matrix (Fig 2B) and admixture proportions (Fig 3C) to our estimates on the real data [59]. Both simulations have n = 1000 individuals, m = 300, 000 loci, and K = 10 subpopulations or intermediate subpopulations. These simulations have FST = 0.1, comparable to previous estimates between human populations (in 1000 Genomes, the estimated FST between CEU (European-Americans) and CHB (Chinese) is 0.106, between CEU and YRI (Yoruba from Nigeria) it is 0.139, and between CHB and YRI it is 0.161 [23]).

10.1371/journal.pgen.1009241.g002 Coancestry matrices of simulations.

Both panels have n = 1000 individuals along both axes, K = 10 subpopulations (final or intermediate), and FST = 0.1. Color corresponds to θjkT between individuals j and k (equal to φjkT off-diagonal, fjT along the diagonal). (A) The independent subpopulations model has θjkT=0 between subpopulations, and varying θjjT per subpopulation, resulting in a block-diagonal coancestry matrix. (B) Our admixture scenario models a 1D geography with extensive admixture and intermediate subpopulation differentiation that increases with distance, resulting in a smooth coancestry matrix with no independent subpopulations (no θjkT=0 between blocks). Individuals are ordered along each axis by geographical position.

 10.1371/journal.pgen.1009241.g003 1D admixture scenario.

We model a 1D geography population that departs strongly from the independent subpopulations model. (A) K = 10 intermediate subpopulations, evenly spaced on a line, evolved independently in the past with FST increasing with distance, which models a sequence of increasing founder effects (from left to right) to mimic the global human population. (B) Once differentiated, individuals in these intermediate subpopulations spread by random walk modeled by Normal densities. (C) n = 1000 individuals, sampled evenly in the same geographical range, are admixed proportionally to the previous Normal densities. Thus, each individual draws most of its alleles from the closest intermediate subpopulation, and draws the fewest alleles from the most distant populations. Long-distance random walks of intermediate subpopulation individuals results in kinship for admixed individuals that decays smoothly with distance in Fig 2B. (D) For FST estimators that require a partition of individuals into subpopulations, individuals are clustered by geographical position (K = 10).

The independent subpopulations simulation satisfies the HudsonK and BayeScan estimator assumptions: each independent subpopulation Su has a different FST value of fSuT relative to the MRCA population T (Fig 2A). Ancestral allele frequencies piT are drawn uniformly between 0.01 and 0.5. Allele frequencies piSu for Su and locus i are drawn independently from the Balding-Nichols (BN) distribution [3] with parameters piT and fSuT. Every individual j in subpopulation Su draws alleles randomly with probability piSu. Subpopulation sample sizes are drawn randomly (Methods, section Simulations).

The admixture simulation corresponds to a “BN-PSD” model [6, 27, 34, 60, 77]: the intermediate subpopulations are independent subpopulations that draw piSu from the BN model, then each individual j constructs its allele frequencies as πij=∑u=1KpiSuqju, which is a weighted average of the subpopulation allele frequencies piSu with the admixture proportions qju of individual j and subpopulation u as weights (which satisfy ∑u=1Kqju=1), as in the Pritchard-Stephens-Donnelly (PSD) admixture model [63–65]. We constructed qju that model admixture resulting from spread by random walk of the intermediate subpopulations along a one-dimensional geography, as follows. Intermediate subpopulations Su are placed on a line with differentiation fSuT that grows with distance, which corresponds to a serial founder effect (Fig 3A). Upon differentiation, individuals in each Su spread by random walk, a process modeled by Normal densities (Fig 3B). Admixed individuals derive their ancestry proportional from these Normal densities, resulting in a genetic structure governed by geography (Figs 3C and 2B) and departing strongly from the independent subpopulations model (Fig 3D). The amount of spread—which sets the mean kinship across all individuals—was chosen to give a bias coefficient of sT=θ¯TFST=0.5, which by Eq (32) results in a large downward bias for F^STstd (in contrast, the independent subpopulations simulation has sT = 0.1). The true coancestry and FST parameters of this simulation are given by the fSuT values of the intermediate subpopulations and the admixture coefficients qju of the individuals via the following equations [57]: θjkT=∑u=1KqjuqkufSuT,FST=∑j=1n∑u=1Kwjqju2fSuT. The first equation above connecting coancestry to admixture proportions was derived independently in other work [62], but the FST for the admixed individuals was absent and instead follows from our generalized FST definition given in Eq (9). See Methods, section Simulations for additional details regarding these simulations.

 Evaluation of <italic>F</italic><sub>ST</sub> estimators

Our admixture simulation illustrates the large biases that can arise if existing FST estimators that require independent subpopulations or FST estimates derived from existing kinship estimators are misapplied to arbitrary population structures to estimate the generalized FST, and demonstrate the higher accuracy of our new FST estimator (F^STnew given by the combination of Eqs (36) and (37)). The WC FIT (total inbreeding) estimator was also evaluated.

First, we test these estimators in our independent subpopulations simulation. The HudsonK (Methods, section Generalized HudsonK FST estimator) and BayeScan FST estimators are consistent in this simulation, since their assumptions are satisfied (Fig 4A). The WC FST estimator assumes that fSuT=FST for all subpopulations Su, which does not hold; nevertheless, WC has only a small bias (Fig 4A). The WC FIT estimator arrives at similar estimates, as it should since there is no local inbreeding, so the true FIT also equals FST. The Weir-Hill estimator permits different fSuT values per subpopulation, but assigns equal weight to individuals rather than subpopulations (Methods, section The Weir-Hill FST estimator), resulting in a slightly different target FST (we verified that these estimates are unbiased for this FST). For comparison, we show the standard kinship-based F^STstd in Eq (25) (weights from Methods, section Simulations) and F^STWG based on the Weir-Goudet kinship estimates for individuals, both of which do not have corrections that would make them consistent under the independent subpopulations model. Since the number of subpopulations K is large, F^STstd has a small relative bias of about sT=1K=10% (Fig 4A); greater bias is expected for smaller K. Our new FST estimator has a very small bias in this simulation resulting from estimating the minimum kinship from the smallest kinship between subpopulations (see Eq (37)) rather than their average as HudsonK does implicitly (Fig 4A).

10.1371/journal.pgen.1009241.g004 Evaluation of <italic>F</italic><sub>ST</sub> estimators.

The Weir-Cockerham, Weir-Hill, Weir-Goudet (for individuals), HudsonK (equal to Weir-Goudet for subpopulations, S1 Text), BayeScan, F^STstd in Eq (25) derived from the standard kinship estimator, and our new FST estimator in Eqs (34) and (37), are evaluated on simulated genotypes from our two models (Fig 2). The Weir-Cockerham FIT estimator was also included to show that estimation of total inbreeding behaves similarly to FST estimators. (A) The independent subpopulations model required by the Weir-Hill, HudsonK, and BayeScan FST estimators. All but standard kinship (F^STstd) and Weir-Goudet (for individuals) recover the target FST IBD probability in Eq (9) (red line) with small errors. (B) Our admixture scenario, which has no independent subpopulations, was constructed so F^STstd≈12FST. Only our new estimates are accurate. The rest of these estimators give values smaller than the target FST IBD probability, which result from treating kinship as zero between every subpopulations imposed by geographic clustering (or between individuals for Standard Kinship and Weir-Goudet). The F^STindep estimator limit in Eq (14) (green dotted line) overlaps the true FST (red line) in (A) but not (B). Estimates (blue) include 95% prediction intervals (often too narrow to see) from 39 independently-simulated genotype matrices for each model (Methods, section Prediction intervals).

Next we test these estimators in our admixture simulation. To apply the FST estimators that require subpopulations to the admixture model, individuals are clustered into subpopulations by their geographical position (Fig 3D). We find that estimates of all existing methods are smaller than the true FST by nearly half, as predicted by the limit of F^STindep in Eq (14) (Fig 4B). The WC FIT estimator obtains slightly larger estimates than the WC FST estimator, but overall remains as biased as the other FST estimators, showing that the use of a total inbreeding estimator for independent subpopulations displays the same bias as the corresponding FST estimator. By construction, the kinship-based F^STstd also has a large relative bias of about sT = 50%; remarkably, all existing FST estimators for subpopulations suffer from comparable biases. Thus, the corrections for independent subpopulations present in the WC, Weir-Hill and HudsonK estimators, or the Bayesian likelihood modeling of BayeScan, are insufficient for accurate estimation of the target generalized FST (Eq (9)) in this admixture scenario. Only our new FST estimator achieves accurate estimates of the generalized FST in the admixture simulation (Fig 4B).

 Evaluation of kinship estimators

Our admixture simulation illustrates the distortions of the standard kinship estimator φ^jkT,std in Eq (18), the linearly-transformed kinship values given by the Weir-Goudet estimator, and demonstrates the improved accuracy of our new kinship estimator φ^jkT,new given by the combination of Eqs (34) and (37). Kinship matrix estimates and their limits are visualized as heatmaps in Fig 5, whereas estimator accuracy is shown directly in Fig 6. The limit of the standard estimator φ^jkT,std in Eq (18) would have had a uniform bias if φ¯jT=φ¯T held for all individuals j. For that reason, our admixture simulation has varying differentiation fSuT per intermediate subpopulation Su (Fig 3A), which causes large differences in φ¯jT per individual j and therefore large distortions in φ^jkT,std. The Weir-Goudet approach estimates the linearly-transformed kinship values calculated in Eq (38).

10.1371/journal.pgen.1009241.g005 Evaluation of kinship estimators.

Observed accuracy for two existing kinship coefficient estimators is illustrated in our admixture simulation and contrasted to the nearly unbiased estimates of our new estimator. Plots show n = 1000 individuals along both axes, and color corresponds to φjkT between individuals j ≠ k and to fjT along the diagonal (fjT is in the same scale as φjkT for j ≠ k; plotting φjjT, which have a minimum value of 12, would result in a discontinuity in this figure). (A) True kinship matrix. (B) Estimated kinship using our new estimator in Eqs (34) and (37) from simulated genotypes recovers the true kinship matrix with high accuracy. (C) Theoretical limit of φ^jkT,std in Eq (19) as the number of independent loci goes to infinity demonstrates the accuracy of our bias predictions under the kinship model. (D) Standard kinship estimates φ^jkT,std given by Eq (18) from simulated genotypes are downwardly biased on average and distorted by pair-specific amounts. (E) Theoretical limit of the Weir-Goudet kinship estimator given by Eq (38). (F) Weir-Goudet kinship estimates from the same simulated genotypes agree with our calculated limit.

 10.1371/journal.pgen.1009241.g006 Accuracy of kinship estimators.

Here the estimated kinship values are directly compared to their true values, in the same admixture simulation data (n = 1000 individuals) shown in the previous figure. (A) Kinship between different individuals (excluding inbreeding). The new estimator has practically no bias in this evaluation (falls on the 1-1 dashed gray line). The standard estimator has a complex, non-linear bias that covers a large area of errors. (B) Inbreeding comparison, shows the bias of the standard estimate follows a different pattern for inbreeding compared to kinship between individuals. To better visualize and compare data across panels, a random subset of n points (out of the original n(n − 1)/2 unique individual pairs) were plotted in (A), matching the number of individuals (number of points in (B)).

Our new kinship estimator (Fig 5B) recovers the true kinship matrix of this complex population structure (Fig 5A), with an RMSE of 2.83% relative to the mean φjkT (Fig 6). In contrast, estimates using the standard estimator have a large overall downward bias (Fig 5C), resulting in an RMSE of 115.72% from the true φjkT relative to the mean φjkT (Fig 6). Additionally, estimates from φ^jkT,std are very distorted, with an abundance of φ^jkT,std<φjkT cases—some of which are negative estimates (blue in Fig 5C)—but remarkably also cases with φ^jkT,std>φjkT (top left corner of Figs 5C and 6).

Now we compare the convergence of the ratio-of-means and mean-of-ratios versions of the standard kinship estimator to their biased limit we calculated in Eq (19) (Fig 5D). The ratio-of-means estimate φ^jkT,std (Fig 5C) has an RMSE of 2.14% from its limit relative to the mean φjkT. In contrast, the mean-of-ratios estimates that are prevalent in the literature have a greater RMSE of 10.77% from the same limit in Eq (19). Thus, as expected from our theoretical results in section Assessing the accuracy of genome-wide ratio estimators, the ratio-of-means estimate is much closer to the desired limit than the mean-of-ratio estimate. The distortions are similar for the estimator that uses IAFs in Eq (24), with reduced RMSEs from its limit of 0.32% and 8.82% for the ratio-of-means and mean-of-ratios estimates, respectively.

 Evaluation of oracle-adjusted <italic>F</italic><sub>ST</sub> estimators

Here we verify additional calculations for the bias of the standard kinship-based estimator F^STstd and the unbiased adjusted “oracle” FST estimators that require the true mean kinship φ¯T or the bias coefficient sT to be known. Note that F^STnew in Eq (36) is related but not identical to these oracle estimators. We tested both IAF (Fig 7A) and genotype (Fig 7B) versions of these estimators. The unadjusted F^STstd in Eq (26) is severely biased (blue in Fig 7) by construction, and matches the calculated limit for IAFs and genotypes (green lines in Fig 7, which are close because φ¯T≈θ¯T). In contrast, the two consistent adjusted estimators F^ST′ and F^ST′′ in Eqs (27) and (31) estimate FST quite well (blue predictions overlap the true FST red line in Fig 7). However, F^ST′ and F^ST′′ are oracle methods, since they require parameters (φ¯T, θ¯T, sT) that are not known in practice.

10.1371/journal.pgen.1009241.g007 Evaluation of standard and adjusted <italic>F</italic><sub>ST</sub> estimators.

The convergence values we calculated for the standard kinship plug-in and adjusted FST estimators are validated using our admixture simulation. All adjusted estimators are unbiased but are “oracle” methods, since the mean kinship (φ¯T), mean coancestry (θ¯T), or bias coefficient (sT=θ¯TFST for IAFs, replaced by φ¯TFST for genotypes) are usually unknown. (A) Estimation from individual-specific allele frequencies (IAFs): F^STstd is the standard coancestry plug-in estimator in Eq (26); F^ST′ “Adj. θ¯T” is in Eq (27); F^ST′′ “Adj. s” is in Eq (31). (B) For genotypes, F^STstd is given in Eq (25), and the adjusted estimators use φ¯T rather than θ¯T. Lines: true FST (red line), limits of biased estimators F^STstd (green lines, which differ slightly per panel). Estimates (blue) include 95% prediction intervals (too narrow to see) from 39 independently-simulated genotype matrices for our admixture model (Methods, section Prediction intervals).

Prediction intervals were computed from estimates over 39 independently-simulated IAF and genotype matrices (Methods, section Prediction intervals). Estimator limits are always contained in these intervals because the number of independent loci (m = 300, 000) is sufficiently large. Estimates that use genotypes have wider intervals than estimates from IAFs; however, IAFs are not known in practice, and use of estimated IAFs might increase noise. Genetic linkage, not present in our simulation, will also increase noise in real data.

 Discussion

We studied analytically the most commonly-used estimators of FST and kinship, which can be derived using the method of moments. We determined the estimation limits of convergence of these approaches under two models of arbitrary population structure (Fig 1). We found that no existing approaches estimate the generalized FST (an IBD probability) accurately (but note that some of these approaches intended to estimate a linearly-transformed FST quantity and not the IBD probability). We also showed that the standard kinship estimator is biased on structured populations (particularly when the average kinship is comparable to the kinship coefficients of interest), and this bias varies for each pair of individuals. These results led us to a new kinship estimator, which is consistent if the minimum kinship is estimated consistently (Fig 1). We presented an implementation of this approach, which is practically unbiased in our simulations. Our kinship and FST estimates in human data are consistent with the African Origins model while suggesting that human differentiation is considerably greater than previously estimated [59].

Estimation of FST in the correct scale is crucial for its interpretation as an IBD probability, for obtaining comparable estimates in different datasets and across species, as well as for DNA forensics [3, 7, 19, 20, 78–80]. Our framework results in a new unbiased genome-wide FST estimator. However, our findings may not have direct implications for single-locus FST estimate approaches where only the relative ranking matters, such as for the identification of loci under selection [8, 10, 81–86], assuming that the bias of the genome-wide estimator carries over uniformly to all single-locus estimates. Our convergence calculations in section Assessing the accuracy of genome-wide ratio estimators require large numbers of loci, so they do not apply to single-locus estimates. Moreover, various methods for single-locus FST estimation for multiple alleles suffer from a strong dependence to the maximum allele frequency and heterozygosity [83–85, 87–90] that suggests that a more complicated bias is present in these single-locus FST estimators.

We have shown that the misapplication of existing FST estimators for independent subpopulations may lead to downwardly-biased estimates that can approach zero even when the true generalized FST is large. Weir-Cockerham [17], Weir-Hill [4], HudsonK (which generalizes the Hudson pairwise FST estimator [23] to K independent populations; also equals the Weir-Goudet approach for subpopulations [21]; S1 Text), and BayeScan [10]FST estimates in our admixture simulation are all smaller than the FST target by nearly a factor of two (Fig 4B), and differ from our new FST estimates in humans by nearly a factor of three [59]. To be accurate, existing FST estimators require independent subpopulations, so the observed biases arise from their misapplication to subpopulations that are neither independent not homogeneous. Nevertheless, natural populations—particularly humans—often do not adhere to the independent subpopulations model [59, 91–95].

The standard kinship coefficient estimator we investigated is often used to control for population structure in GWAS and to estimate genome-wide heritability [16, 27, 30–35]. While this estimator was known to be biased [16, 35], no closed-form limit had been calculated until very recently [21, 62]. These kinship estimates are biased downwards on average, but bias also varies for each pair of individuals (Figs 1 and 5). Thus, the use of these distorted kinship estimates may be problematic in GWAS or for estimating heritability, but the extent of the problem remains to be determined.

We developed a theoretical framework for assessing genome-wide ratio estimators of FST and kinship. We proved that common ratio-of-means estimators converge almost surely to the ratio of expectations for infinite independent loci (S1 Text). Our result justifies approximating the expectation of a ratio-of-means estimator with the ratio of expectations [4, 17, 23]. However, mean-of-ratios estimators may not converge to the ratio of expectations for infinite loci. Mean-of-ratios estimators are potentially asymptotically unbiased for infinite individuals, but it is unclear which estimators have this behavior. We found that the ratio-of-means kinship estimator had much smaller errors from the ratio of expectations than the more common mean-of-ratios estimator, whose convergence value is unknown. Therefore, we recommend ratio-of-means estimators, whose asymptotic behavior is well understood.

Our new framework enables accurate FST estimation in more complex datasets than before, but challenges remain. One challenge is the estimation of local inbreeding coefficients, which are required for estimating the generalized FST when not all individuals are locally outbred. To this end, we suggest employing existing approaches that infer inbreeding from large runs of homozygosity or related strategies [66–68], particularly when such self-IBD blocks are much larger than observed between individuals in the same subpopulation. A streamlined approach for jointly estimating total and local inbreeding is desirable, but will require an appropriate evaluation featuring realistic simulation of local inbreeding in a complex population structure. Another challenge is the estimation of the minimum kinship value without the use of subpopulation labels, so that accurate FST estimates can be obtained with even less user supervision. A more general unsupervised method could better ensure accuracy under extreme cases, such as when there are few unrelated individual pairs. These challenges can be overcome with the estimators we have presented, although supervision is needed to ensure that local inbreeding and the minimum kinship are estimated correctly.

We have demonstrated the need for new models and methods to study complex population structures, and have proposed a new approach for kinship and FST estimation that provides nearly unbiased estimates in this setting. Extending our implementation to deliver consistent accuracy in arbitrary population structures will require further innovation, and the results provided here may be useful in leading to more robust estimators in the future.

 Methods Previous <italic>F</italic><sub>ST</sub> estimators for the independent subpopulations model

Here we summarize the previous Weir-Cockerham, Weir-Hill, and Hudson FST estimators for independent subpopulations and derive the generalized HudsonK estimator for more than two subpopulations (which also equals the recent Weir-Goudet FST estimator for subpopulations under biallelic loci; S1 Text). We show that each of these estimators reduces, under infinite subpopulation sizes, to F^STindep in Eqs (11) to (13) that was studied in the results. In this section, let i index the m loci, j index the n subpopulations, nj be the number of individuals sampled from subpopulation j, and p^ij be the sample reference allele frequency at locus i in subpopulation j.

The Weir-Cockerham <italic>F</italic><sub>ST</sub> estimator

The Weir-Cockerham (WC) FST estimator [17] estimates the coancestry parameter θT shared by each of the n independent subpopulation in consideration. Let h^ij denote the fraction of heterozygotes in subpopulation j for locus i. The ratio-of-means WC FST estimator and its limit for independent subpopulations (θjkT=0 for j ≠ k) with equal differentiation (θjjT=θT) is n¯=1n∑j=1nnj,C2=1n¯2(n-1)∑j=1n(nj-n¯)2,p^iT=1n∑j=1nnjn¯p^ij,h¯i=1n∑j=1nnjn¯h^ij,σ^i2=1n-1∑j=1nnjn¯(p^ij-p^iT)2,F^STWC=∑i=1mσ^i2-1n¯-1(p^iT(1-p^iT)-n-1nσ^i2-14h¯i)∑i=1mp^iT(1-p^iT)(1-n¯C2n(n¯-1))+1nσ^i2(1+(n-1)n¯C2n(n¯-1))+h¯iC24n(n¯-1)→m→∞a.s.FST=θT. Note that p^iT above weighs every individual equally by weighing subpopulation j proportional to its sample size nj, so it equals the estimator in Eq (17) with uniform weights.

Now we simplify this estimator as the sample size of every subpopulation becomes infinite. First set the sample size of every subpopulation nj equal to their mean n¯, which implies C2 = 0 and p^iT=1n∑j=1np^ij,h¯i=1n∑j=1nh^ij,σ^i2=1n-1∑j=1n(p^ij-p^iT)2,F^STWC=∑i=1mσ^i2-1n¯-1(p^iT(1-p^iT)-n-1nσ^i2-14h¯i)∑i=1mp^iT(1-p^iT)+1nσ^i2. Now we take the limit as the sample size n¯→∞, which results in sample allele frequencies converging to the true subpopulation allele frequencies p^ij→πij for every subpopulation j and locus i, and p^iT=1n∑j=1nπij,σ^i2=1n-1∑j=1n(πij-p^iT)2,F^STWC=∑i=1mσ^i2∑i=1mp^iT(1-p^iT)+1nσ^i2, which matches the F^STindep in Eqs (11) to (13) as desired. Note the number of subpopulations n remains finite, and the sample heterozygosity h¯i is not needed in the limit.

 The Weir-Hill <italic>F</italic><sub>ST</sub> estimator

Weir and Hill developed new estimators for subpopulation-specific FST values and considered the effects of non-independent subpopulations [4]. However, these estimators target linearly-transformed FST values, and recover the FST defined in Eq (9) only when subpopulations are independent [4], so we group them here with other estimators that strictly assume independent subpopulations. For simplicity, here we only consider the global FST estimator; the estimators of the coancestry matrix of the subpopulations was found to have the same overall linear transformation [4]. In the limit of infinite subpopulation sizes, this estimator also converges to the asymptotic FST estimator for independent subpopulations (F^STindep) discussed in the main text.

The Weir-Hill (WH) FST estimator, simplified here for biallelic loci but extended to average over loci, and its limit, are given by p^iT=∑j=1nwjp^ij,wj=nj∑j=1nnj,F^STWH=1-(∑j=1nnj(1-wj))(∑i=1m∑j=1nwj2nj2nj-1p^ij(1-p^ij))∑i=1m∑j=1nnj(p^ij-p^iT)2+nj(1-wj)p^ij(1-p^ij)→m→∞a.s.FST-θ˜T1-θ˜T, where the target FST and θ˜T both weigh individuals (rather than subpopulations) equally [4]: FST=∑j=1nwjθjjT,θ˜T=21-∑j=1nwj2∑j=2n∑k=1j-1wjwkθjkT. For equal sample sizes nj = nS∀j, we have wj=1n, njc=nS(1-1n), and the estimator becomes p^iT=1n∑j=1np^ij,σ^i2=1n-1∑j=1n(p^ij-p^iT)2,F^STWH=∑i=1mσ^i2(2nS-1n2ns-1)-piT(1-piT)(12ns-1)∑i=1mpiT(1-piT)+1nσ^i2. Therefore, as sample sizes per subpopulation go to infinity (nS → ∞, which results in p^ij→πij for every (i, j)), we again recover the desired limiting FST estimator for independent subpopulations (F^STindep in Eqs (11) to (13)).

 The Hudson <italic>F</italic><sub>ST</sub> estimator

The Hudson pairwise FST estimator [23] measures the differentiation of two subpopulations (j, k). The estimator and its limit for two independent subpopulations (θjkT=0) is F^STHudson=∑i=1m(p^ij-p^ik)2-p^ij(1-p^ij)2nj-1-p^ik(1-p^ik)2nk-1∑i=1mp^ij(1-p^ik)+p^ik(1-p^ij)→m→∞a.s.FST=θjjT+θkkT2.

Generalized HudsonK <italic>F</italic><sub>ST</sub> estimator

Here we derive the “HudsonK” estimator (first made available in [58]), which generalizes the Hudson pairwise FST estimator in Eq (40) to n independent subpopulations. This estimator also equals the recent Weir-Goudet FST estimator for subpopulations [21] (for biallelic loci; S1 Text). Note that for independent subpopulations, the FST of all the subpopulations equals the mean pairwise FST of every pair of subpopulations: 1n2∑j=1n∑k=1n(θjjT+θkkT2)=1n∑j=1nθjjT=FST. For that reason, averaging numerators and denominators of the pairwise estimator in Eq (40) before computing the ratio, we obtain the generalized estimator and a limit under independent subpopulations of p^iT=1n∑j=1np^ij,σ^i2=1n-1∑j=1n(p^ij-p^iT)2,F^STHudsonK=∑i=1mσ^i2-1n∑j=1np^ij(1-p^ij)2nj-1∑i=1mp^iT(1-p^iT)+1nσ^i2→m→∞a.s.FST=1n∑j=1nθjjT. Note that unlike the WC and Weir-Hill estimators, p^iT above weighs every subpopulation equally, so every individual is weighed inversely proportional to the sample sizes nj of their subpopulation j.

Like WC and Weir-Hill, F^STHudsonK simplifies to F^STindep in Eqs (11) to (13) in the limit of infinite sample sizes nj → ∞, where p^ij→πij for every (i, j).

 Simulations Construction of subpopulation allele frequencies

We simulate K = 10 subpopulations Su and m = 300, 000 independent loci. Every locus i draws piT~Uniform(0.01,0.5). We set fSuT=uKτ, where τ ≤ 1 tunes FST. For the independent subpopulations model, FST=1K∑u=1KfSuT=τ(K+1)2K, so τ=2KFSTK+1 gives the desired FST (τ ≈ 0.18 for FST = 0.1). For the admixture model, τ is found numerically (τ ≈ 0.90 for FST = 0.1; see last subsection). Lastly, piSu values are drawn from the Balding-Nichols distribution, piSu|T∼Beta(piT(1fSuT-1),(1-piT)(1fSuT-1)), which results in subpopulation allele frequencies that obey the coancestry model of Eq (6), with E[piSu|T]=piT and Var(piSu|T)=fSuTpiT(1-piT) [3], as desired.

 Random subpopulation sizes

We randomly generate sample sizes r = (ru) for K subpopulations and ∑u=1Kru=n=1000 individuals, as follows. First, draw x ∼ Dirichlet (1, …, 1) of length K and r = round(n x). While minuru<n3K, draw a new r, to prevent small subpopulations (they do not occur in real data). Due to rounding, ∑u=1Kru may not equal n as desired. Thus, while δ=n-∑u=1Kru≠0, a random u is updated to ru ← ru + sgn(δ), which brings δ closer to zero at every iteration. Weights for individuals j in Su are wj=1Kru so the generalized FST matches FST=1K∑u=1KfSuT from the independent subpopulations model (see section The generalized FST for arbitrary population structures), which HudsonK estimates.

 Admixture proportions from 1D geography

We construct qju from random-walk migrations along a one-dimensional geography. Let xu be the coordinate of intermediate subpopulation u and yj the coordinate of a modern individual j. We assume qju is proportional to f(|xu − yj|), or qju=f(|xu-yj|)∑v=1Kf(|xv-yj|). where f is the Normal density function with μ = 0 and tunable σ. The Normal density models random walks, where σ sets the spread of the populations (Fig 5). Our simulation uses xu = u and yj=12+j-1n-1K, so the intermediate subpopulations span between 1 and K and individuals span between 12 and K+12. For the FST estimators that require subpopulations, individual j is assigned to the nearest subpopulation Su (the u that minimizes |xu − yj|; Fig 3D); these subpopulations have equal sample size, so wj=1n is appropriate.

 Choosing <italic>σ</italic> and <italic>τ</italic>

Here we find values for σ (controls qjk) and τ (scales fSuT) that give sT=12 and FST = 0.1 in the admixture model. In our simulation, wj=1n and fSuT=uKτ, so applying those parameters to Eq (39) gives θjkT=τK∑u=1Kuqjuqku and FST=τnK∑j=1n∑u=1Kuqju2. Therefore, sT=θ¯TFST=1n∑u=1Ku(∑j=1nqju(σ))2∑u=1Ku(∑j=1nqju2(σ)) depends only on σ. A numerical root finder finds that σ ≈ 1.78 gives sT=12. For fixed qju, τ=FST1K∑u=1Ku(1n∑j=1nqju2). FST = 0.1 is achieved with τ ≈ 0.901.

 Prediction intervals

Prediction intervals with α = 95% correspond to the range of n = 39 independent FST estimates. In the general case, n independent statistics are given in order X(1) < … < X(n). Then I = [X(j), X(n+1−j)] is a prediction interval with confidence α=n+1-2jn+1 [96]. In our case, j = 1 and n = 39 gives α = 0.95, as desired. Each estimate was constructed from simulated data with the same dimensions and structure as before (fixed fSuT and qju; fixed sample sizes for the independent subpopulations model), but with piT,piSu,πij,xij drawn separately for each estimate.

 BayeScan and Weir-Goudet implementations

Weir-Goudet (WG) kinship estimates [20–22] were calculated using the function snpgdsIndivBeta in the R package SNPRelate 1.20.1 available on Bioconductor and GitHub. We found identical estimates using the function beta.dosage in the R package hierfstat 0.4.30 available on GitHub. WG (individuals) FST estimates were computed from the kinship estimates as described in section Comparison to the Weir-Goudet kinship estimator for individuals.

BayeScan 2.1 was downloaded from http://cmpg.unibe.ch/software/BayeScan/. To estimate FST, first the per-subpopulation FST values were estimated across loci assuming no selection, then the global FST was given by the mean FST across subpopulations.

 Software

An R package called popkin, which implements the kinship and FST estimation methods proposed here, is available on the Comprehensive R Archive Network (CRAN) at https://cran.r-project.org/package=popkin and on GitHub at https://github.com/StoreyLab/popkin.

An R package called bnpsd, which implements the BN-PSD admixture simulation, is available on CRAN at https://cran.r-project.org/package=bnpsd and on GitHub at https://github.com/StoreyLab/bnpsd.

An R package called popkinsuppl, which implements memory-efficient algorithms for the Weir-Cockerham, Weir-Hill, and HudsonK FST estimators, and the standard kinship estimator, is available on GitHub at https://github.com/OchoaLab/popkinsuppl.

Public code reproducing these analyses are available at https://github.com/StoreyLab/human-differentiation-manuscript.

 Supporting information Supplementary information.

Includes mathematical proofs and other calculations, including proof of convergence of ratio-of-means estimators, proof that the Weir-Goudet FST estimator for subpopulations equals HudsonK, derivation of existing method-of-moment estimators, proof that FST and kinship estimator limits are constants with respect to the ancestral population T, mean coancestry bounds, moments of estimator building blocks, the derivation of our new kinship estimator, and proof that our estimator from our original 2016 manuscript is algebraically equivalent to the one presented here.

(PDF)

 References Malécot G. Mathématiques de l’hérédité. Masson et Cie; 1948. Wright S. The genetical structure of populations. Ann Eugen. 1951;15(4):323–354. Balding DJ, Nichols RA. A method for quantifying differentiation between populations at multi-allelic loci and its implications for investigating identity and paternity. Genetica. 1995;96(1-2):3–12. doi: 10.1007/BF01441146 Weir BS, Hill WG. Estimating F-Statistics. Annual Review of Genetics. 2002;36(1):721–750. doi: 10.1146/annurev.genet.36.050802.093940 Nicholson G, Smith AV, Jónsson F, Gústafsson O, Stefánsson K, Donnelly P. Assessing population differentiation and isolation from single-nucleotide polymorphism data. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2002;64(4):695–715. doi: 10.1111/1467-9868.00357 Falush D, Stephens M, Pritchard JK. Inference of population structure using multilocus genotype data: linked loci and correlated allele frequencies. Genetics. 2003;164(4):1567–1587. Balding DJ. Likelihood-based inference for genetic correlation coefficients. Theoretical Population Biology. 2003;63(3):221–230. doi: 10.1016/S0040-5809(03)00007-8 Beaumont MA, Balding DJ. Identifying adaptive genetic divergence among populations from genome scans. Molecular Ecology. 2004;13(4):969–980. doi: 10.1111/j.1365-294X.2004.02125.x Foll M, Gaggiotti O. Identifying the Environmental Factors That Determine the Genetic Structure of Populations. Genetics. 2006;174(2):875–891. doi: 10.1534/genetics.106.059451 Foll M, Gaggiotti O. A genome-scan method to identify selected loci appropriate for both dominant and codominant markers: a Bayesian perspective. Genetics. 2008;180(2):977–993. doi: 10.1534/genetics.108.092221 Coop G, Witonsky D, Rienzo AD, Pritchard JK. Using Environmental Correlations to Identify Loci Underlying Local Adaptation. Genetics. 2010;185(4):1411–1423. doi: 10.1534/genetics.110.114819 Thompson EA. The estimation of pairwise relationships. Ann Hum Genet. 1975;39(2):173–188. doi: 10.1111/j.1469-1809.1975.tb00120.x Milligan BG. Maximum-likelihood estimation of relatedness. Genetics. 2003;163(3):1153–1167. Jacquard A. Structures génétiques des populations. Paris: Masson et Cie; 1970. Csűrös M. Non-identifiability of identity coefficients at biallelic loci. Theor Popul Biol. 2014;92:22–29. doi: 10.1016/j.tpb.2013.11.001 Astle W, Balding DJ. Population Structure and Cryptic Relatedness in Genetic Association Studies. Statist Sci. 2009;24(4):451–471. doi: 10.1214/09-STS307 Weir BS, Cockerham CC. Estimating F-Statistics for the Analysis of Population Structure. Evolution. 1984;38(6):1358–1370. doi: 10.1111/j.1558-5646.1984.tb05657.x Weir BS, Cardon LR, Anderson AD, Nielsen DM, Hill WG. Measures of human population structure show heterogeneity among genomic regions. Genome Res. 2005;15(11):1468–1476. doi: 10.1101/gr.4398405 Buckleton J, Curran J, Goudet J, Taylor D, Thiery A, Weir BS. Population-specific FST values for forensic STR markers: A worldwide survey. Forensic Science International: Genetics. 2016;23:91–100. doi: 10.1016/j.fsigen.2016.03.004 Weir B, Zheng X. SNPs and SNVs in forensic science. Forensic Science International: Genetics Supplement Series. 2015;5(Dec):e267–e268. Weir BS, Goudet J. A Unified Characterization of Population Structure and Relatedness. Genetics. 2017;206(4):2085–2103. doi: 10.1534/genetics.116.198424 Goudet J, Kay T, Weir BS. How to estimate kinship. Mol Ecol. 2018;27(20):4121–4135. doi: 10.1111/mec.14833 Bhatia G, Patterson N, Sankararaman S, Price AL. Estimating and interpreting FST: the impact of rare variants. Genome Res. 2013;23(9):1514–1521. doi: 10.1101/gr.154831.113 Xie C, Gessler DD, Xu S. Combining different line crosses for mapping quantitative trait loci using the identical by descent-based variance component method. Genetics. 1998;149(2):1139–1146. Yu J, Pressoir G, Briggs WH, Vroh Bi I, Yamasaki M, Doebley JF, et al. A unified mixed-model method for association mapping that accounts for multiple levels of relatedness. Nat Genet. 2006;38(2):203–208. doi: 10.1038/ng1702 16380716 Aulchenko YS, de Koning DJ, Haley C. Genomewide rapid association using mixed model and regression: a fast and simple method for genomewide pedigree-based quantitative trait loci association analysis. Genetics. 2007;177(1):577–585. doi: 10.1534/genetics.107.075614 Price AL, Patterson NJ, Plenge RM, Weinblatt ME, Shadick NA, Reich D. Principal components analysis corrects for stratification in genome-wide association studies. Nat Genet. 2006;38(8):904–909. doi: 10.1038/ng1847 Kang HM, Zaitlen NA, Wade CM, Kirby A, Heckerman D, Daly MJ, et al. Efficient control of population structure in model organism association mapping. Genetics. 2008;178(3):1709–1723. doi: 10.1534/genetics.107.080101 18385116 Kang HM, Sul JH, Service SK, Zaitlen NA, Kong SY, Freimer NB, et al. Variance component model to account for sample structure in genome-wide association studies. Nat Genet. 2010;42(4):348–354. doi: 10.1038/ng.548 20208533 Zhou X, Stephens M. Genome-wide efficient mixed-model analysis for association studies. Nat Genet. 2012;44(7):821–824. doi: 10.1038/ng.2310 Yang J, Benyamin B, McEvoy BP, Gordon S, Henders AK, Nyholt DR, et al. Common SNPs explain a large proportion of the heritability for human height. Nat Genet. 2010;42(7):565–569. doi: 10.1038/ng.608 20562875 Yang J, Lee SH, Goddard ME, Visscher PM. GCTA: a tool for genome-wide complex trait analysis. Am J Hum Genet. 2011;88(1):76–82. doi: 10.1016/j.ajhg.2010.11.011 Rakovski CS, Stram DO. A kinship-based modification of the armitage trend test to address hidden population structure and small differential genotyping errors. PLoS ONE. 2009;4(6):e5825. doi: 10.1371/journal.pone.0005825 Thornton T, McPeek MS. ROADTRIPS: case-control association testing with partially or completely unknown population and pedigree structure. Am J Hum Genet. 2010;86(2):172–184. doi: 10.1016/j.ajhg.2010.01.001 Speed D, Balding DJ. Relatedness in the post-genomic era: is it still useful? Nat Rev Genet. 2015;16(1):33–44. doi: 10.1038/nrg3821 Wang B, Sverdlov S, Thompson E. Efficient Estimation of Realized Kinship from SNP Genotypes. Genetics. 2017; p. genetics.116.197004. doi: 10.1534/genetics.116.197004 Wright S. Systems of Mating. V. General Considerations. Genetics. 1921;6(2):167–178. doi: 10.1093/genetics/6.2.167 Lush JL. Heritability of Quantitative Characters in Farm Animals. Hereditas. 1949;35(S1):356–375. doi: 10.1111/j.1601-5223.1949.tb03347.x Falconer DS, Mackay TFC. Introduction to Quantitative Genetics. 4th ed. Harlow: Pearson; 1996. Thompson EA. Identity by descent: variation in meiosis, across genomes, and in populations. Genetics. 2013;194(2):301–326. doi: 10.1534/genetics.112.148825 Slatkin M. Inbreeding coefficients and coalescence times. Genetics Research. 1991;58(2):167–175. doi: 10.1017/S0016672300029827 Emik LO, Terrill CE. Systematic procedures for calculating inbreeding coefficients. J Hered. 1949;40(2):51–55. doi: 10.1093/oxfordjournals.jhered.a105986 García-Cortés LA. A novel recursive algorithm for the calculation of the detailed identity coefficients. Genetics Selection Evolution. 2015;47(1):33. doi: 10.1186/s12711-015-0108-6 Rosenberg NA, Pritchard JK, Weber JL, Cann HM, Kidd KK, Zhivotovsky LA, et al. Genetic Structure of Human Populations. Science. 2002;298(5602):2381–2385. doi: 10.1126/science.1078311 12493913 Ramachandran S, Deshpande O, Roseman CC, Rosenberg NA, Feldman MW, Cavalli-Sforza LL. Support from the relationship of genetic and geographic distance in human populations for a serial founder effect originating in Africa. Proc Natl Acad Sci U S A. 2005;102(44):15942–15947. doi: 10.1073/pnas.0507611102 Consortium TGP. A map of human genome variation from population-scale sequencing. Nature. 2010;467(7319):1061–1073. doi: 10.1038/nature09534 Lazaridis I, Patterson N, Mittnik A, Renaud G, Mallick S, Kirsanow K, et al. Ancient human genomes suggest three ancestral populations for present-day Europeans. Nature. 2014;513(7518):409–413. doi: 10.1038/nature13673 25230663 Lazaridis I, Nadel D, Rollefson G, Merrett DC, Rohland N, Mallick S, et al. Genomic insights into the origin of farming in the ancient Near East. Nature. 2016;536(7617):419–424. doi: 10.1038/nature19310 27459054 Skoglund P, Posth C, Sirak K, Spriggs M, Valentin F, Bedford S, et al. Genomic insights into the peopling of the Southwest Pacific. Nature. 2016;538(7626):510–513. doi: 10.1038/nature19844 27698418 Tishkoff SA, Reed FA, Friedlaender FR, Ehret C, Ranciaro A, Froment A, et al. The Genetic Structure and History of Africans and African Americans. Science. 2009;324(5930):1035–1044. doi: 10.1126/science.1172257 19407144 Moreno-Estrada A, Gravel S, Zakharia F, McCauley JL, Byrnes JK, Gignoux CR, et al. Reconstructing the Population Genetic History of the Caribbean. PLOS Genetics. 2013;9(11):e1003925. doi: 10.1371/journal.pgen.1003925 24244192 Moreno-Estrada A, Gignoux CR, Fernández-López JC, Zakharia F, Sikora M, Contreras AV, et al. The genetics of Mexico recapitulates Native American substructure and affects biomedical traits. Science. 2014;344(6189):1280–1285. doi: 10.1126/science.1251688 24926019 Leslie S, Winney B, Hellenthal G, Davison D, Boumertit A, Day T, et al. The fine-scale genetic structure of the British population. Nature. 2015;519(7543):309–314. doi: 10.1038/nature14230 25788095 Baharian S, Barakatt M, Gignoux CR, Shringarpure S, Errington J, Blot WJ, et al. The Great Migration and African-American Genomic Diversity. PLoS Genet. 2016;12(5):e1006059. doi: 10.1371/journal.pgen.1006059 27232753 Haak W, Lazaridis I, Patterson N, Rohland N, Mallick S, Llamas B, et al. Massive migration from the steppe was a source for Indo-European languages in Europe. Nature. 2015;522(7555):207–211. doi: 10.1038/nature14317 25731166 Allentoft ME, Sikora M, Sjögren KG, Rasmussen S, Rasmussen M, Stenderup J, et al. Population genomics of Bronze Age Eurasia. Nature. 2015;522(7555):167–172. doi: 10.1038/nature14507 26062507 Ochoa A, Storey JD. FST and kinship for arbitrary population structures I: Generalized definitions. bioRxiv. 2016; doi: 10.1101/083915 Ochoa A, Storey JD. FST and kinship for arbitrary population structures II: Method of moments estimators. bioRxiv. 2016; doi: 10.1101/083923 Ochoa A, Storey JD. New kinship and FST estimates reveal higher levels of differentiation in the global human population. bioRxiv. 2019; doi: 10.1101/653279 Thornton T, Tang H, Hoffmann TJ, Ochs-Balcom HM, Caan BJ, Risch N. Estimating kinship in admixed populations. Am J Hum Genet. 2012;91(1):122–138. doi: 10.1016/j.ajhg.2012.05.024 Hao W, Song M, Storey JD. Probabilistic models of genetic variation in structured populations applied to global human studies. Bioinformatics. 2016;32(5):713–721. doi: 10.1093/bioinformatics/btv641 Zheng X, Weir BS. Eigenanalysis of SNP data with an identity by descent interpretation. Theoretical Population Biology. 2016;107:65–76. doi: 10.1016/j.tpb.2015.09.004 Pritchard JK, Stephens M, Donnelly P. Inference of population structure using multilocus genotype data. Genetics. 2000;155(2):945–959. Tang H, Peng J, Wang P, Risch NJ. Estimation of individual admixture: analytical and study design considerations. Genet Epidemiol. 2005;28(4):289–301. doi: 10.1002/gepi.20064 Alexander DH, Novembre J, Lange K. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 2009;19(9):1655–1664. doi: 10.1101/gr.094052.109 Browning BL, Browning SR. A Fast, Powerful Method for Detecting Identity by Descent. The American Journal of Human Genetics. 2011;88(2):173–182. doi: 10.1016/j.ajhg.2011.01.010 Gazal S, Sahbatou M, Perdry H, Letort S, Génin E, Leutenegger AL. Inbreeding Coefficient Estimation with Dense SNP Data: Comparison of Strategies and Application to HapMap III. HHE. 2014;77(1-4):49–62. Joshi PK, Esko T, Mattsson H, Eklund N, Gandin I, Nutile T, et al. Directional dominance on stature and cognition in diverse human populations. Nature. 2015;523(7561):459–462. doi: 10.1038/nature14618 26131930 Cochran WG. Sampling techniques. 3rd ed. Wiley; 1977. Reynolds J, Weir BS, Cockerham CC. Estimation of the Coancestry Coefficient: Basis for a Short-Term Genetic Distance. Genetics. 1983;105(3):767–779. doi: 10.1093/genetics/105.3.767 Weir BS. Genetic data analysis II. Methods for discrete population genetic data. Sunderland, USA: Sinauer Associates; 1996. Bourgain C, Hoffjan S, Nicolae R, Newman D, Steiner L, Walker K, et al. Novel case-control test in a founder population identifies P-selectin as an atopy-susceptibility locus. Am J Hum Genet. 2003;73(3):612–626. doi: 10.1086/378208 12929084 Choi Y, Wijsman EM, Weir BS. Case-Control Association Testing in the Presence of Unknown Relationships. Genet Epidemiol. 2009;33(8):668–678. doi: 10.1002/gepi.20418 Pickrell JK, Pritchard JK. Inference of population splits and mixtures from genome-wide allele frequency data. PLoS Genet. 2012;8(11):e1002967. doi: 10.1371/journal.pgen.1002967 Nei M. Analysis of Gene Diversity in Subdivided Populations. PNAS. 1973;70(12):3321–3323. doi: 10.1073/pnas.70.12.3321 Weir BS, Goudet J. A unified characterization of population structure and relatedness. bioRxiv. 2016; p. 088260. Raj A, Stephens M, Pritchard JK. fastSTRUCTURE: variational inference of population structure in large SNP data sets. Genetics. 2014;197(2):573–589. doi: 10.1534/genetics.114.164350 Nelis M, Esko T, Mägi R, Zimprich F, Zimprich A, Toncheva D, et al. Genetic Structure of Europeans: A View from the North–East. PLOS ONE. 2009;4(5):e5472. doi: 10.1371/journal.pone.0005472 19424496 Silva NM, Pereira L, Poloni ES, Currat M. Human Neutral Genetic Variation and Forensic STR Data. PLOS ONE. 2012;7(11):e49666. doi: 10.1371/journal.pone.0049666 Steele CD, Court DS, Balding DJ. Worldwide FST Estimates Relative to Five Continental-Scale Populations. Annals of Human Genetics. 2014;78(6):468–477. doi: 10.1111/ahg.12081 Cavalli-Sforza LL. Population Structure and Human Evolution. Proceedings of the Royal Society of London Series B, Biological Sciences. 1966;164(995):362–379. Lewontin RC, Krakauer J. Distribution of Gene Frequency as a Test of the Theory of the Selective Neutrality of Polymorphisms. Genetics. 1973;74(1):175–195. doi: 10.1093/genetics/74.1.175 Beaumont MA, Nichols RA. Evaluating Loci for Use in the Genetic Analysis of Population Structure. Proceedings of the Royal Society of London B: Biological Sciences. 1996;263(1377):1619–1626. doi: 10.1098/rspb.1996.0237 Vitalis R, Dawson K, Boursot P. Interpretation of Variation Across Marker Loci as Evidence of Selection. Genetics. 2001;158(4):1811–1823. Akey JM, Zhang G, Zhang K, Jin L, Shriver MD. Interrogating a high-density SNP map for signatures of natural selection. Genome Res. 2002;12(12):1805–1814. doi: 10.1101/gr.631202 Porter AH. A test for deviation from island-model population structure. Molecular Ecology. 2003;12(4):903–915. doi: 10.1046/j.1365-294X.2003.01783.x Bowcock AM, Kidd JR, Mountain JL, Hebert JM, Carotenuto L, Kidd KK, et al. Drift, admixture, and selection in human evolution: a study with DNA polymorphisms. PNAS. 1991;88(3):839–843. doi: 10.1073/pnas.88.3.839 1992475 Hedrick PW. A Standardized Genetic Differentiation Measure. Evolution. 2005;59(8):1633–1638. doi: 10.1111/j.0014-3820.2005.tb01814.x Jakobsson M, Edge MD, Rosenberg NA. The Relationship Between FST and the Frequency of the Most Frequent Allele. Genetics. 2013;193(2):515–528. doi: 10.1534/genetics.112.144758 Edge MD, Rosenberg NA. Upper bounds on FST in terms of the frequency of the most frequent allele and total homozygosity: the case of a specified number of alleles. Theor Popul Biol. 2014;97:20–34. Lewontin RC. The Apportionment of Human Diversity. Evolutionary Biology. 1972;6:381–398. Barbujani G, Magagni A, Minch E, Cavalli-Sforza LL. An apportionment of human DNA diversity. PNAS. 1997;94(9):4516–4519. doi: 10.1073/pnas.94.9.4516 Novembre J, Johnson T, Bryc K, Kutalik Z, Boyko AR, Auton A, et al. Genes mirror geography within Europe. Nature. 2008;456(7218):98–101. doi: 10.1038/nature07331 18758442 Coop G, Pickrell JK, Novembre J, Kudaravalli S, Li J, Absher D, et al. The Role of Geography in Human Adaptation. PLoS Genet. 2009;5(6):e1000500. doi: 10.1371/journal.pgen.1000500 19503611 Patterson N, Moorjani P, Luo Y, Mallick S, Rohland N, Zhan Y, et al. Ancient admixture in human history. Genetics. 2012;192(3):1065–1093. doi: 10.1534/genetics.112.145037 22960212 Beran R, Hall P. Interpolated Nonparametric Prediction Intervals and Confidence Intervals. Journal of the Royal Statistical Society Series B (Methodological). 1993;55(3):643–652. doi: 10.1111/j.2517-6161.1993.tb01929.x