We analysed gene variant data from a total of 7595 individuals from the NHLBI GO Exome Sequencing Project (ESP) [16] and the 1000 Genomes Project [21]. We considered 17764 stop-gain and 13915 frameshift variants collectively affecting 11369 autosomal protein coding genes reliably annotated by the Consensus CDS (CCDS) project [23]. The distributions of gene truncating variants according to allele frequency and study are presented in Table S1.

Consistent with previous reports [19], [24], we observed that the distribution of stop-gain and frameshift variants along the protein coding sequence of genes is biased by allele frequency (Figure S1). Variants with very low allele frequency (MAF≤0.001) are evenly distributed, with a modest 3′ terminal enrichment. However, the distribution of stop-gain and frameshift variants becomes less uniform with increasing allele frequencies, yet does not show a clear pattern. In contrast, we observed marked distribution trends in association with the following sequence features: (i) loss of functional domains; (ii) disruption of constitutive exons (i.e. exons present in all isoforms), or of principal isoforms; (iii) localization in potential NMD-targeted regions. In comparison to rare truncating variants, common stop-gain and frameshift variants were clearly depleted at positions leading to the loss of a functional domain (Figure 1A). Analysis of splicing-dependent effects was limited to genes with multiple annotated transcripts in CCDS (n = 5203). We observed an enrichment of common stop-gain and frameshift variants in alternative isoforms (Figure 1B) and a depletion of common variants in principal isoforms (Figure 1C). Defining principal isoform on the basis of highest expression level across tissues [25] showed comparable results. We observed that common gene truncating variants occurred less frequently in regions more than fifty nucleotides upstream the last exon-exon junction, possibly triggering NMD-mediated transcript degradation (Figure 1D). For all the features discussed above, gene-truncating variants associated with disease in the Online Mendelian Inheritance in Man (OMIM) database exhibited a distribution bias opposite to what was observed for common stop-gain and frameshift variants (Figure 1). The same trends were observed when ESP and 1000 Genomes variant datasets were analysed separately (Figure S2).

We used expression data from 421 individuals to assess the functional impact of stop-gain and frameshift variants [22]. In particular, we evaluated differences between protein truncating variants localized to NMD-targeted region compared to those that were not. Stop-gains predicted to trigger NMD (n = 756) had a significantly lower expression level (median Z-score = −0.59) than stop-gains predicted to escape NMD (n = 379, median = −0.10) and lower than a reference distribution of synonymous variants (median = −0.04, one-sided Wilcoxon rank-sum test p-value<2.2e-16) (Figure 2A). Among stop-gains predicted to trigger NMD, singletons (n = 488, median Z-score = −0.75) showed a stronger decrease in expression level compared to non-singletons (n = 268, median = −0.26, p-value = 1.3e-10, Figure 2B), which is an indication that they represent actual variants and not sequencing or bioinformatics errors. We did not observe a similar reduction when considering 87 of the 172 frameshift variants with expression data mapping to potential NMD-target regions (median = −0.14, p-value = 5.7e-02).

To further evaluate a splicing-dependent impact on gene expression levels, we limited the analysis to 301 stop-gains predicted to trigger NMD and affecting genes with multiple isoforms described in CCDS. We observed a significant decrease in gene expression levels of NMD-triggering stop-gains affecting all isoforms (n = 216, median = −0.64) compared to those affecting only a fraction of isoforms (n = 85, median = −0.22, one-sided Wilcoxon rank-sum test p-value = 2.5e-03) (Figure 2C). Similar results were obtained using RPKM normalized expression values (Figure S3). These observations confirmed the functional impact of stop-gains consistently with predictions of degradation by NMD and current annotation of isoforms.

We then evaluated the predictive value for pathogenicity of the sequence-based features characterized in the previous sections: percentage of sequence affected, loss of functional domains, proportion of isoforms affected, principal isoform damage, and NMD-target region. We integrated them into a naïve Bayes classifier (Table S3) and assessed its performance over a dataset of 1160 pathogenic stop-gain variants found in the OMIM database and 125 common stop-gain variants that are not known to be pathogenic. Predictive performance of the pathogenicity score was validated over unseen variants excluded from the learning data using successive random subsampling (see Methods). The classifier was benchmarked against a state of the art gene-based probability score proposed by MacArthur et al [19]. This gene-based score relies on conservation and protein interaction network proximity to genes associated to a recessive disease as predictive features. In the case of stop-gain variants, the performance of the gene-based method was consistent with the reported results in the original work (Area Under the Curve (AUC) = 0.83, Figure 3A). Similar ROC curves were obtained with the gene-based score RVIS [6] that provides a measure of the departure from the average number of common functional mutations in genes with a similar amount of mutational burden (Figure S4). The score based on sequence features alone showed a lower predictive value (AUC = 0.67). However, optimal ROCs were achieved by combining sequence and gene-based scores (Figure 3). We observe that at a False Positive Rate (FPR) of less than 0.1 there is no improvement from the combined sequence-based and from the MacArthur gene-based score that used network proximity OMIM recessive disease genes in its design. Improvement at low FPR occurs in the combination of the sequence-based score with RVIS, which does not rely on OMIM annotations. While the AUC improvement is modest, it is consistent across two datasets (ESP and 1000 Genomes), over the two gene-based scores, and for the two types of variants (stop-gains and frameshifts), Figure S4. These results demonstrate that sequence features can be incorporated as an additional source of information to improve current pathogenicity prediction.

The marginal improvement obtained when scores were combined motivated us to explore whether the different approaches were capturing independent information. We observed a very low correlation between gene-based and sequence-based scores (Spearman rank correlation <0.13, p-value: <2.2e-16 ([0.10,0.13] 95% CI from 10,000 bootstrap samples). The reason for this observation is that the various scores are based on different criteria: gene conservation and centrality (MacArthur 2012), burden of variation (RVIS) and sequence features (current work). Correlations were not increased in analyses limited to OMIM disease variants (Figure S5). Based on these results we explored the potential for complementarity across scores.

First, we analysed whether the sequence-based score was better powered to detect functional impact as measured by effect on gene expression. We observed a stronger correlation with expression levels for the sequence-based score (Spearman rank correlation = 0.21±0.03, p-value: <5e-12) than either gene-based scores (0.06±0.04, p-value>0.05 for MacArthur 2012 score and 0.13±0.03, p-value<5e-05, for RVIS score), Figure 4. Second, we analysed OMIM genes that carry variants annotated as pathogenic in OMIM as well as unknown or non-pathogenic variants. Here, the variants are scored differently using a sequence-based approach, while all share the same gene-based score. Figure 5 depicts this situation for 95 OMIM disease genes carrying multiple stop-gains. The genes with the highest pathogenicity gene-based scores also carried variants with very low severity as determined by a sequence-based score. Third, we checked whether the performance of the sequence-based score varies depending on the degree of gene conservation, as measured by dN/dS ratio in the same set of OMIM disease genes. Figure 6 shows that, for genes below the protein-coding genome average dN/dS (0.261), the MacArthur and RVIS gene-based scores resulted in higher pathogenicity estimates than the sequence-based score; however without discriminating between pathogenic and non-pathogenic/non-annotated variants. In contrast, for genes with dN/dS≥0.261, the sequence-based score performed similarly for pathogenic variants while attributing less pathogenicity to non-pathogenic/non-annotated variants of the same gene (Wilcoxon signed rank test p-value<0.012). We note that OMIM variants used here were not considered for learning in the Bayesian classification (see Methods).

From these results, we conclude that the two types of scores are complementary. Specifically, the sequence-based score improves measurement of functional gene impairment, discriminates across different variants in a given gene and appears particularly useful for analysis of less conserved genes.

To test the ability to rank the functional consequences of gene truncating variants in innate immunity genes, we analysed the distribution of both the sequenced-based and gene-based pathogenicity scores in 1503 genes involved in innate immunity [20], including 387 interferon stimulated genes (ISGs, [26] [27]). We identified 856 innate immunity genes, including 230 ISGs, carrying rare gene truncating variants (MAF<1%). Globally, innate immunity and OMIM genes ranked higher than the background set of the genome for both scores (Figure 7 and Figure S6). However, the highest scores were obtained for stop-gain variants in OMIM genes, particularly for variants in innate immunity genes that are not observed in the ESP or the 1000 Genomes Project samples (Figure 7). The latter result is consistent with their extreme rarity and severity. We note that OMIM variants used here for validation were not considered for learning in the Bayesian classification (see Methods). Despite their apparent agreement in Figure 7, correlation between the sequenced-based and gene-based pathogenicity scores was very low (Spearman correlation below 0.31 in all sets of genes analyzed) indicating that both scores provide complementary information.

Given the observation that truncating variants can be associated with important differences in functional impact, we estimated the number of individuals in the study population that carried variants consistently annotated as highly pathogenic by one or several scores. Among 7595 individuals and 1503 innate immunity genes, 33 individuals carried rare (MAF<0.01) stop-gains and 85 carried rare frameshifts that scored with high severity (pathogenicity rank percentile < = 20%) in all scores (sequence-based, MacArthur 2012 and RVIS). For the smaller set of 387 ISGs, we identified 8 individuals carrying rare stop-gains and 4 carrying rare frameshifts with high severity in all scores.

We then focused on truncating variants in genes associated with viral inhibition in cellular assays [26], [27]. A total of 13 out of 42 genes carried such variants (observed in at least 2 people), which were very rare overall (MAF<0.0053; Table S4). Specifically, two genes had variants with high predicted pathogenicity based on both scores: MX1 which controls Influenza A virus in vitro and HPSE which is involved in metapneumovirus, respiratory syncytial virus and yellow fever virus control. While the gene-based scores were by definition identical for all variants affecting a same gene, the sequenced-based score sharply distinguished the variants according to different predictive pathogenicity (Table S3). This observation was consistent with the observed differences in gene expression levels available for some of the variants.

Two genetic variant and annotation datasets were used: 1) 6503 individuals from the NHLBI GO Exome Sequencing Project (ESP) [16] and 2) 1092 individuals from the 1000 Genomes Project [21]. Variants (SNPs and INDELs) and annotations for the ESP exomes (file ESP6500SI-V2-SSA137.dbSNP138-rsIDs.snps_indels.txt.tar.gz) were downloaded from the Exome Variant Server, NHLBI GO Exome Sequencing Project, Seattle, WA (http://evs.gs.washington.edu/EVS/, accessed July 2013). Only variants assigned to the following categories were considered for further analysis: “stop-gained” (including “stop-gained-near-splice”), “frameshift”, “coding-synonymous” (including “coding-synonymous-near-splice”) and “missense” (including “missense-near-splice”). One base was added to the genomic coordinates reported for frameshifts in the ESP dataset to consider the actual location of the insertion/deletion event (http://evs.gs.washington.edu/EVS/HelpDescriptions.jsp?tab=tabs-1). Variants and genotypes from the 1000 Genome Project [21] correspond to phase 1 version 3 of the 20110521 release (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20110521/, accessed August 2013). SnpEff Variant Analysis software [32] (version 3.3h build 2013-08-11) was used to annotated 1000Genome variants against SnpEff's pre-built human database (GRCh37.71). SnpEff categories labeled with errors or warnings in the EFF field were disregarded. Only variants assigned to the following categories were considered for further analysis: “stop_gained”, “frame_shift”, “synonymous_coding” (including “synonymous_start” and “synonymous_stop”) and “non_synonymous_coding” (missense). Hardy-Weinberg equilibrium (HWE) was tested with R package GWASExactHW (http://cran.r-project.org/web/packages/GWASExactHW/, version 1.1). A fraction of variants significantly deviated from HWE (Fisher's exact p-values<0.05), mainly due to an excess of homozygous rare allele calls, likely indicating technical artifacts. All variants not in HWE were filtered out. When both datasets where considered together, the following criteria were adopted: i) Genomic coordinates of frameshift variants reported by both datasets were treated as reported for the ESP dataset. ii) Allele frequencies and HWE of variants present in both datasets were derived from the sum of individuals from both studies; allele frequencies of variants present in only one dataset were taken as originally reported by the corresponding dataset. To exclude bias due to previous assumptions, results were reproduced for the two datasets considered separately as well as combined. For the combined analysis allele frequencies of variants present in only one dataset are estimated over all 7593 individuals.

The analysis pipeline implemented to annotate genetic variants is depicted in Figure S7. We restricted the analysis to protein coding genes and transcripts annotated by the Consensus CDS (CCDS) project [23] (ftp.ncbi.nlm.nih.gov/pub/CCDS/, Release 12 04/30/2013). We considered only variants affecting a core set of human protein coding regions consistently annotated and of high quality. Only genes on the 22 autosomes were retained and only CCDS entries with a public status and an identical match were kept.

Domains of human protein sequences were retrieved from the InterPro database [33] (release 44.0, 23/09/2013). Data were downloaded through BioMart Central Portal [34] (http://central.biomart.org/, accessed 04/10/2013), filtering fragments and considered domain boundaries corresponding to InterPro “supermatches”. Mapping from InterPro coordinates on UniProt protein sequences to CCDS sequences was done by exact matching of the complete amino acid sequences using UniProt database (release 2013_07; [35]).

A position within a protein coding gene was considered alternatively spliced if it was shared by only a fraction of all protein coding transcripts reported by the Consensus CCDS Project for that gene. Otherwise it was considered constitutively spliced for the purpose of the study. Annotation of principal isoforms used APPRIS ([36]; file APPRIS-g15.v3.15Jul2013/appris_data.principal.homo_sapiens.tsv accessed 03/09/2013 at URL: http://appris.bioinfo.cnio.es), a computational pipeline and database for annotations of human splice isoforms. APPRIS selects a specific transcript as principal isoform, i.e. the one computationally predicted as responsible of the main cellular function, being expressed in most of the tissues or developmental stages and more evolutionary conserved. Selection of the principal isoform is based on protein structure, function and interspecies conservation of transcripts. As an alternative definition of principal isoform, we identified the transcript with a recurrent highest expression level across tissues as provided by [25].

We accounted for nonsense-mediated decay (NMD) following HAVANA annotation guidelines v.20 (05/04/2012) (http://www.sanger.ac.uk/research/projects/vertebrategenome/havana/assets/guidelines.pdf), Specifically, the NMD-target region of a transcript was defined as those positions more than 50 nucleotides upstream the 3′-most exon-exon junction. Transcripts bearing stop-gain variants at these regions are predicted to be degraded by NMD [13].

Geuvadis RNA sequencing data from 421 lymphoblastoid cell lines from the 1000 Genomes Project (phase 1 version 3 of the 20110521 release, see above; [21]) were obtained from Lappalainen et al. 2013 [22]. Gene expression quantifications of protein-coding genes were downloaded from EBI ArrayExpress accession E-GEUV-1 (accessed 05/11/2013). Analyses were independently performed on both RPKM and Peer-factor normalized RPKM values. As a measure of the impact of a variant on expression level, we calculated the average Z-score of the expression level in cells from individuals carrying the variant compared to all samples.

We used a naïve Bayes classification scheme in order to derive a probability of pathogenicity for a given variant using the following sequence-based features: maximum transcript length affected, maximum percentage of domain truncation, number of isoforms and ratio of isoforms affected, truncation of the principal isoform and localization in an NMD-target region. Solely for the purpose of the classifier, missing values were imputed to zero for percentage of domain truncation and to the longest isoform for principal isoform annotation. We defined a matrix X_N×K of K sequence-based features for N variants of a given type in the dataset and a binary vector c_N×1 annotating variants as benign or pathogenic. A new variant, y_1×K, is evaluated using maximum likelihood estimates for class-specific means from the annotated data, and a common intra-class variance vector (except for binary features). We estimate the variance vector as ν = E[(x_i - μ_ci)²], where x_i is the i^th row in matrix X and μ_ci is the mean vector corresponding to the class indicated by c_i,. We assigned a pathogenic class with 1 and benign class with 0. Assuming a prior probability of pathogenicity, p₁, posterior probability of pathogenicity can be evaluated as:where p₀ = 1-p₁ is the prior probability of being benign and θ = {μ₁,μ₂,ν} is the set of model parameter vectors. The conditional likelihood of y for a given class is assumed to factorize as product of K likelihoods corresponding to the K sequence features available (naïve Bayes assumption). We used normal, and Bernoulli likelihood functions to model continuous and binary features respectively. It is straightforward to show the ranking produced from this posterior probability does not depend on the prior probability p₁ as long as it is larger than zero and it is equal for all the mutations under consideration.

As reference throughout the work, and as a learning set for the predictive scores (ROC analyses), we used a catalogue of pathogenic mutations from the Online Mendelian Inheritance in Man [37] database. Only genes with a cytogenetic location (genemap2.txt accessed 18/10/2013 at OMIM: ftp.omim.org) and with a gene status of confirmed or provisional were kept. For each gene with an associated OMIM number, all allelic variants with a “live” status and a dbSNP identifier were obtained through the OMIM API server (http://api.omim.org/). We used Ensembl Variation [38] (Ensembl release 71, April 2013, dataset Homo sapiens Short Variation, SNPs and indels, GRCh37.p10, accessed 25/10/2013 at http://apr2013.archive.ensembl.org/biomart/martview/) to obtain the genomic coordinates for each dbSNP identifier together with the clinical significance of each specific allele as reported by ClinVar and dbSNP following OMIM guidelines (http://www.ncbi.nlm.nih.gov/clinvar/docs/clinsig/). Only variants with a dbSNP identifier annotatted as “pathogenic” and mapping to a unique genomic location were kept for further analysis. SnpEff Variant Analysis was then used to re-annotate the selected pathogenic variants as described above.

We benchmarked three different pathogenicity scores using all stop-gain variants from the OMIM dataset as positive (pathogenic), and all common variants (MAF≥1%) not present in OMIM dataset as negative (benign) variants. The sequence-based score is the posterior probability calculated from the naïve Bayes classification scheme described in previous section using an empirical prior for pathogenicity. We used two different gene-based scores: first the probability provided by MacArthur et al [19] for prioritization of variants derived from two gene-level features: conservation and protein interaction network proximity to genes associated with a recessive disease. And second the Residual Variation Intolerance Score (RVIS) [6] that provides a measure of the departure from the average number of common functional mutations in genes with a similar amount of mutational burden. RVIS pathogenic score was assessed as f(-RVIS), where f(.) is the logistic function. The joint score was defined as the product of the sequence-based score and one of the two previously defined gene-based scores. This joint score can be interpreted as the joint probability of a pathogenic mutation in a gene assuming conditional independence of the two probability scores. The receiver operating characteristic (ROC) curve was derived using random subsampling validation iterations. In each iteration, we use 75% of the data to train the classifier and use the remaining 25% for validation. This was done 10000 times, to minimize the Monte Carlo error, and validation set scores were combined to calculate the ROC curve. The same procedure was applied to frameshift variants.

Except for ROC analyses, sequence-based scores used throughout the work were derived from the learning set described above excluding both i) OMIM pathogenic variants reported in ESP and 1000 Genomes Project and ii) OMIM pathogenic variants affecting innate immunity genes and interferon stimulated genes.

Genome-wide codon alignments of orthologous genes for nine primate species (human, chimpanzee, gorilla, orangutan, macaque, marmoset, tarsier, bushbaby, and mouse lemur) were collected from Ensembl v57. We assessed dN/dS estimates using both Ensembl Compara's protein-based alignments, and DNA-based alignments of primate sequences generated from genomic DNA alignments. Sitewise Likelihood Ratio test [39] was used to calculate the overall dN/dS for a given gene based on a one-ratio model where all sites have the same dN/dS value.

A representative list of 1503 human innate immunity genes [20] was used. Within this list, we further analyzed 387 interferon stimulated genes (ISGs) [26]. Additionally, we focused on those ISGs showing antiviral activity against 18 viruses (including important human pathogens such as HIV-1, hepatitis C virus, influenza virus and other respiratory viruses) upon overexpression in in vitro cellular assays [26], [27]. We first identified all ISGs carrying gene truncating variants, and then characterized the subset of those genes associated with more than 50% viral inhibition in the cellular assays.