All the experiments reported in this paper use a gene model trained on a nonredundant set of 3,038 single-gene sequences. We built this set by combining the Augustus training set [2], the GenScan training set, and 1,500 high-confidence CDSs from EnsMart Plus [22], which are part of the Genezilla training set (http://www.tigr.org/software/traindata.shtml). We then appended simulated intergenic material to both ends of each training sequence to make up for the lack of realistic intergenic regions in the training material, as described in more detail in Methods.

We compared CRAIG with GenScan, TwinScan 2.03 (without homology features, also known as GenScan++), Genezilla (formerly known as TigrScan), and Augustus on several benchmark test sets. We also ran predictions with HMMGene, the only other publicly available genefinder to use a discriminative structure training method; we present some prediction results with it in Methods. All programs we compare with are based on similar GHMM models with similar sequence features. Augustus uses two types of length distributions for introns: short intron lengths are modeled with an explicit distribution, but other introns use the default geometric distribution. This difference made Augustus run many times slower than the other programs in all our experiments.

We evaluated the programs on the following benchmark test sets.

Table 2 shows prediction results for all programs on BGHM957. CRAIG achieved better sensitivity and specificity than the other programs at all levels, except for somewhat lower base sensitivity but much higher base specificity than GenScan. The relative F-score improvement for initial and single exons over Genezilla, the second-best program overall for this set, was 14.6% and 5.8%, respectively. Single-exon genes were more difficult to predict for all programs, with specificity barely exceeding 50% for the best program, but CRAIG's relative improvement in sensitivity was nearly 25% over runner-up Genezilla. Terminal exon predictions were also improved over the nearest competitors, but less markedly so. The improved gene-level accuracy follows from these gains at the exon level. GenScan++ and Augustus predicted internal exons with similar accuracy and their F-scores were only slightly worse than CRAIG, but the overall gene-level accuracy for GenScan++ looks much worse because it missed many terminal and single exons. GenScan also did well in this set, but overall performance was somewhat worse than the other programs.

Most of the genes in this set have short introns and the intergenic regions are truncated, so prediction was relatively easy and all programs did relatively well. The next section compares performance on datasets with long-intron genes and very long intergenic regions.

As previously noted, TIGR251 has many genes with very long introns, so it is expected to be harder to predict accurately. This was confirmed by the results in Table 3. Performance was worse for all programs and levels when compared with the first set. However, CRAIG consistently outperformed the other programs with an even wider performance gap than in the first experiment. Here, base and internal-exon accuracies were also substantially improved. CRAIG's relative F-score improvement for bases and internal exons over Genezilla, the second-best program in both categories, was 5.4% and 7.1%, respectively, compared with approximately 1% for BGHM953. Other types of exons also improved, as in the first experiment. Because of these better base and exon-level predictions, the relative F-score improvement over runner-up Genezilla at the gene level was about 57%.

Our final set of experiments was on ENCODE294. The results are shown in Table 4. As previously mentioned, all sequences in this set were masked for low-complexity regions and repeated elements. Unlike previous sets, in which masking did not affect results significantly, prediction on unmasked sequences in this set was worse for all programs (unpublished data). In particular, exon and base specificity decreased an average of 8%.

We added a transcript-level prediction category to Table 4 to better evaluate predictions on alternatively spliced genes. We closely followed the evaluation guidelines and definitions by Guigo and Reese [28]. There, transcript and gene-level predictions that are consistent with annotated incomplete transcripts are counted correct, even in cases where the predictions include additional exons. We relaxed this policy to also mark as correct those predictions that contained incomplete transcripts whose first (last) exon did not begin (end) with an acceptor (donor). The reason for this change is that no program can exactly predict both ends of such transcripts. We developed our own programs to evaluate single-exon, transcript, and gene-level predictions for incomplete transcripts. Evaluations for other categories and for complete transcripts were handled directly with eval.

To ensure consistency in the evaluation, we obtained all of the programs except for Genezilla from their authors and we ran them on the test set in our lab. Genezilla predictions for this set were obtained directly from the supplementary material provided by [28] so that we could measure the potential differences between our evaluation method and that reported in [28], particularly at the transcript and gene level, for which we expected different results.

Overall, our results for all programs agree with those of Guigo and Reese [28]. Genezilla's base and exon-level results using our evaluation program closely matched the published values. Transcript and gene-level results computed by our method were 1% better than the published numbers, which roughly match the percentage of incomplete annotated transcripts with no splice signals on either end. Computed predictions for GenScan and Augustus were also somewhat different, but not substantially so, from those reported by Guigo and Reese [28], presumably because of differences in program version and operating parameters.

Improvements in this set were similar to those obtained in our second experiment. The relative F-score improvements for individual bases and internal exons were 6% and 15.4% over GenScan++ and Augustus, the runner-ups in each respective category. Improvement in prediction accuracy on single, initial, and terminal exons is similar to that for the other test sets. Transcript and gene-level accuracies were, respectively, 30% and 30.6% better than Augustus, the second-best program overall. This means that our better accuracy results obtained in the first two single-gene sequence sets scale well to chromosomal regions with multiple, alternatively spliced genes.

In all tests and at all levels, CRAIG achieved greater improvements in specificity than in sensitivity. We investigated whether the improvements in exon sensitivity achieved by CRAIG could be explained by chance. Any exon belonging to a particular test set is associated with two dependent Bernoulli random variables for whether it was correctly predicted by CRAIG and by another program. We computed p-values with McNemar's test for dependent, paired samples from CRAIG and each of the other programs over the three test sets, as shown in Table 5. The null hypothesis was that CRAIG's advantage in exon predictions is due to chance. The p-values were <0.05 for all entries, except for the TIGR251 experiments against Genezilla and the ENCODE294 experiments against Genezilla and GenScan; in general, these two genefinders proved to be very sensitive at the cost of predicting many more false positives. p-Values for the combined test sets were all below 0.001, showing that CRAIG's advantage was extremely unlikely to be a chance event.

We also trained and tested an additional variant of CRAIG, in which we did not distinguish between short and long introns; this configuration corresponds closely to the state model representation used in most previous works. Following Stanke and Waack [2], we used the relative mean improvement: as the measure of differences in prediction accuracy between the CRAIG variant and CRAIG itself. The term ΔSn_exon denotes the mean increase in exon sensitivity and is defined as where n_t is the number of annotated genes in dataset t, T = {BGHM953, TIGR251, ENCODE294}, and ΔSn^t_exon is the difference in exon sensitivity between CRAIG and the CRAIG variant on dataset t. The other terms are defined similarly. The improvement obtained by CRAIG with respect to the variant was r = 3.6. This result was as expected: there was an improvement in accuracy from including the extra intron state in the gene model, but even the simpler variant was more than competitive with the best current genefinders.

In what follows, a gene structure consists of either a single exon or a succession of alternating exons and introns, trimmed from both ends at the TIS and stop signals. We distinguish two different types of introns: short—980 bp or less—and long—more than 980 bp. Figure 5 shows a gene finite-state model that implements these distinctions.

In what follows, x = x₁…x_P is a sequence and s = s₁…s_Q is a segmentation of x, where each segment s_j = 〈p_j,l_j,y_j〉 starts at position pos(s_j) = p_j, has length len(s_j) = l_j, and state label lab(s_j) = y_j, with p_j+1 = p_j + l_j ≤ P and 1 ≤ l_j ≤ B for some empirically determined upper bound B. The training data. consists of pairs of a sequence and its preferred segmentation. For DNA sequences, x_i ∈ Σ_DNA = {A, T, G, C}, and each label lab(s_j) is one of the states of the model (Figure 5). A segment is also referred to as a genomic region; that is, an exon, an intron, or an intergenic region.

A first-order Markovian linear structure model computes the score of a candidate segmentation s = s₁…s_Q of a given input sequence x as a linear combination of terms for individual features of a candidate segment, the label of its predecessor, and the input sequence. More precisely, each proposed segment s_j is represented by a feature vector f(s_j,lab(s_j−1),x) ∈ ℜ^D computed from the segment, the label of the previous segment, and the input sequence around position pos(s_j). A weight vector_, w ∈ ℜ^D, to be learned, represents the relative weights of the features. Then, the score of candidate segmentation s for sequence x is given by

For gene prediction, we need to answer three basic questions. First, given a sequence, x, we need to efficiently find its best-scoring segmentation. Second, given a training set , we need to learn weights w such that the best-scoring segmentation of x^(t) is close to s^(t). Finally, we need to select a feature function f that is suitable for answering the first two questions while providing good generalization to unseen test sequences. The next three subsections answer these questions.

Let GEN(x) be the set of all possible segmentations of x. The best segmentation of x for weight vector w is given by:

We can compute ŝ efficiently from x using the following Viterbi-like recurrence:

It is easy to see that where GEN_i,y(x) is the set of all segmentations of x₁…x_i that end with label y. Therefore, S_w(x,ŝ) = M(P + 1,END), where END is a special synchronization state inserted at position P + 1. The actual segmentations are easily obtained by keeping back-pointers from each state-position pair (y,i) to its optimal predecessor (y′,i−l). The complexity of this algorithm is O(PBm²), where m is the number of distinct states and B is the upper bound on the segment length, because the runtime of w · f is independent of P, B, or m. To reduce the constant factor from these dot product computations, most w · f values are precomputed and cached. For introns and intergenic regions, the feature function f is a sum of per-nucleotide contributions, so the dynamic program in Equation 4 needs only to look at position i − 1 when y corresponds to such regions. Therefore, B needs to be only the upper bound for exon lengths, which was chosen following Stanke and Waack [2]. For long sequences, the complexity of the inference algorithm is therefore dominated by the sequence length P.

Online learning is a simple, scalable, and flexible framework for training linear structured models. Online algorithms process one training example at a time, updating the model weights to improve the model's accuracy on that example. Large-margin classifiers, such as the well-known SVMs, provide strong theoretical classification error bounds that hold well in practice for many learning tasks. MIRA [30] is an online method for training large-margin classifiers that is easily extended to structured problems [19]. Algorithm 1 shows the pseudocode for the MIRA-based training algorithm we used for our models. For each training sequence, x^(t), the algorithm seeks to establish a margin between the score of the correct segmentation and the score of the best segmentation according to the current weight vector that is proportional to the mismatch between the candidate segmentation and the correct one. MIRA keeps the norm of the change in weight vector as small as possible while giving the current example (x^(t),s^(t)) a score that exceeds that of the best-scoring incorrect segmentation by a margin given by the mismatch between the correct segmentation and the incorrect one. The quadratic program in line 5 of Algorithm 1 formalizes that objective, and has a straightforward closed-form solution for this version of the algorithm. Line 11 of the algorithm computes w as an average of the weight vectors obtained at each iteration, which has been shown to reduce weight overfitting [31]. The training parameter N is determined empirically using an auxiliary development set.

Algorithm 1. Online Training Algorithm.

Training data . L(s^(t),ŝ) is some nonnegative real-valued function that measures the mismatch between segmentation ŝ and the correct segmentation s^(t). The number of rounds N is determined using a small development set.

Successful discriminative learning depends on having training data with statistics similar to the intended test data. However, this is not the case for gene training data. The main distribution mismatch is that reliable gene annotations available for training are for the most part for single-gene sequences with very small flanking intergenic regions.

To address this problem, we created long training sequences composed of actual genes separated by synthetic intergenic regions as follows. For each training sequence, we generated two extra intergenic regions and appended them to both sequence ends, making sure that the total length of both flanking intergenic regions followed geometric distributions with means 5,000, 10,000, 60,000, and 150,000 bp for each of four GC content classes, respectively [3,10]. The synthetic intergenic regions were generated by sampling from GC-dependent, fourth-order interpolated Markov models (IMMs), with the same form as the models we used to score the intergenic state.

Algorithm 1 also requires a loss function, L, and a small development set on which to estimate the number of rounds, N. As loss function, we used the correlation coefficient at the base level [24], since it combines specificity and sensitivity into a single measure. The development set consisted of the 65 genes previously used in GenScan [1] to cross-validate splice signal detectors.

The final ingredient of the CRAIG model is the feature function f used to score candidate segments based on properties of the input sequence. A typical feature relates a proposed segment to some property of the input around that segment, and possibly to the label of the previous segment.

Properties. We started by introducing basic sequence properties that features are based on. These properties are real-valued functions of the input sequence around a particular position. Some properties represent tests, taking the binary values 1 for true and 0 for false. For any test P,‖P‖ denotes the function with value 1 if the test is true, 0 otherwise.

The tests check whether substring u occurs at position i ∈ x. For example, x = ATGGCGGA would have sub_A(1,x) = 1, sub_TA(2,x) = 0, and sub_GGC(3,x) = 1.

The property score_y(i,x) computes the score of a content model for state y at position i. This score is the probability that nucleotide i has label y according to a k-order interpolated Markov model [32], where k = 8 for coding states and k = 4 for noncoding states.

The property gcc(i,x) calculates the GC composition for the region containing position i, averaged over a 10,000-bp window around position i.

Each feature associates a property to a particular model state or state transition.

Binning. Properties with multimodal or sparse distributions, such as segment length, cannot be used directly in a linear model, because their predictive effect is typically a nonlinear function of their value. To address this problem, we binned each property by splitting its range into disjoint intervals or bins, and converting the property into a set of tests that checked whether the value of the property belonged to the corresponding interval. The effect of this transformation was to pass the property through a stepwise constant nonlinearity, each step corresponding to a bin, where the height of each step was learned as the weight of a binary feature associated to the appropriate test.

For example, following GenScan [1], we mapped the GC content property gcc to four bins: <43, 43–51, 51–57, and >57. For other properties, we used regular bins with a property-specific bin width. For instance, exon length was mapped to 90 bp–wide bins.

Test and feature combinations. We used Boolean combinations of tests and binary features to model complex dependencies on the input. Conjunctions can model nucleotide correlations, for example donors of the form G₋₁G₅, that is, donors with G at positions −1 and 5. Likewise, disjunctions were used to model consensus sequences, for example, donors of the form U₃, that is, donors with either an A or a G at position 3.

In general, for two binary functions f and g, we denoted their conjunction by f ∧ g and their disjunction by f ∨ g.

State features encode the content properties of the genomic regions associated to states: exons, introns, and intergenic regions. State features do not depend on the previous state, so we omitted the previous state argument in these feature definitions.

Coding/noncoding potential. This feature corresponds to the log of the probability assigned to the region by the content scoring model: where μ_y is the arithmetic mean of the distribution of log score_y on the training data. For coding regions, the sum is computed over codon scores instead of base scores. Other features related to log score_y also included in f are the coding differential and the score log-ratios between intronic and intergenic regions.

Phase biases. Biases in intron and exon phase distributions have been found and analyzed by Fedorov et al. [33]. We represented possible biases with the straightforward functions where p = 0,1,2 is a phase and I_p is the corresponding intronic state.

Length distributions. The length distributions of exons and introns have been extensively studied. Raw exon lengths were binned to allow our linear model to learn the length histogram from the training data. For long introns, with length >980, we used 980/len(s) as the length feature, whereas shorter introns used max {245/len(s),1}.

For each genomic region type, we also provided length-dependent default features whose weights expressed a bias for or against regions of that length and type. The value of these features is len(s)/λ_y, where λ_y is the average length of all y-labeled segments. For introns and intergenic regions, we used separate, always-on default features for the four classes of GC content discussed above.

Coding composition. In addition to coding potential scores, which give broad, smoothed statistics for different genomic region types, we also defined count features for each 3-gram (codon) and 6-gram (bicodon) in an exon, and similar count features for the first 15 bases (five codons) of an initial exon. The 3-gram features were further split by GC content class. The general form of such a feature is where p = 0,1,2 is the phase, u is the n-gram, and m is the window size, which is len(s) for a general exon count, and min{len(s),15} for special initial exon features, which attempt to capture composition regularities right after the TIS.

Masking. We represented the presence of tandem repeats and other low complexity regions in exonic segments by the function: After training, this feature effectively penalizes any exon whose fraction of N occurrences exceeds 50% of its total length.

Table 6 shows all the state features associated with each segment label.

Transition features look at biological signals that indicate a switch in genomic region type. Features testing for those signals looked for combinations of particular motifs within a window centered at a given offset from the position where the transition occurs. Features of the following form, which test for motif occurrence, are the building blocks for the transition features: where p is the offset, w is the window width, and u is the motif. This feature counts the number of occurrences of u within p ± w/2 bases of the start of segments.

In principle, all sequence positions are potential signal occurrences, but in practice one might filter out unlikely sites, using a sensitivity threshold proportional to level of signal conservation, thus decreasing decoding time.

Burge and Karlin [1] model positional biases within signals with combinations of position weight matrices (PWMs) and their generalizations, weight array models (WAMs) and windowed weight array models (WWAMs), with very good results. It is straightforward to define these models as sets of features based on our motif_p,u,w feature, as shown here in the WWAM case:

PWMs and WAMs are special cases of WWAMs and can thus be defined by PWM_q,r = WWAM_1,1,q,r and WAM_q,r = WWAM_1,2,q,r. This means that we can use all of these techniques to model biological signals in CRAIG with the added advantage of having all signal model parameters trained as part of the gene structure.

Correlations between two positions within a signal are captured by conjunctions of motif features. For example, the feature conjunction would be 1 whenever there is an A in position −3 and a T in position 2 relative to a donor signal occurring at position 156 in x, and 0 otherwise.

We can also extend the feature conjunction operator to sets of features: if A and B are sets of features, such as the WWAM defined in Equation 5, we can define the set of features A ∧ B = {f ∧ g : f ∈ A, g ∈ B}.

In Equation 5, if we use the amino acid alphabet ∑_AA instead of ∑_DNA, and work with codons instead of single nucleotides, we can model signal peptide regions. If we sum over disjunctions of motif features, we can easily model consensus sequences.

Table 7 shows the motif feature sets for each biological signal. The parameters required by each feature type were either taken from the literature [1] or by search on the development set.

We included an additional feature set, motivated by previous work [2], to learn splice site information from sequences that only contain intron annotations. For any donor (acceptor), we first counted the number of similar donors (acceptors) in a given list of introns. A signal was considered to be similar to another if the Hamming distance between them was at most 1. The features were induced by a logarithmic binning function applied over the total number of similarity counts for Hamming distances 0 and 1.