Our data mining strategy can be divided into two parts. In the first part (Figure 1), semi-automated crawlers collected public data from diverse sources, such as NCBI GEO [12], Array Express [13], SMD [14], Broad Cancer Genomics [15], Cancer Biomedical Informatics Grid (caBIG), and other repositories (Table S1). A data analysis step produced sets of differentially expressed gene signatures associated with each experimental or clinical comparison, such as disease versus normal (Methods). In the final step of part one, all signatures were tagged with relevant ontology terms (Figure 1) that reflected associated tissue types, disease/phenotype, compound treatment, or genetic perturbation (e.g., gene mutation, knockout, siRNA knockdown). In the second part, rank-based enrichment statistics were applied to compute pairwise correlation scores between all signatures (Figures 2 and 3) followed by a meta-analysis to compute individual signature-ontology concept correlation scores (Figure 4).

To ensure that data were comparable across different platforms and species, gene signature identifiers were translated using both a universal gene dictionary to map them to a standard NCBI gene reference and a cross-organism dictionary to assign them to precomputed ortholog clusters (Methods). Gene annotations and probe definitions were regularly revised to ensure that they are up-to-date with manufacturer specifications. While issues have been reported with microarray spot definitions [16]–[18], we believe that our meta-analysis approach mitigated the effects of errors that occur on a single platform since the strength of an association was weighted by its consistency across multiple studies and platforms.

We collected and analyzed over 6,000 individual experiments from different public sources of large-scale experimental data. Within this collection were more than 140,000 individual samples profiled on gene expression microarrays. Out of 6,000 experiments, only 4,000 passed our extensive quality control (QC) criteria. Approximately 60% of the disqualified studies were excluded from processing due to insufficient replicates, lack of control samples, or unsupported platforms (e.g., platforms that do not cover more than half the number of genes for an organism). Approximately 25% of the disqualified studies were duplicated (e.g. part of an already processed super-series), and 15% were excluded for failing QC metrics during pre-processing and differential expression analysis (Table 1, Methods).

After applying a statistical analysis to identify differentially expressed genes in each experiment, we obtained a total of 25,000 gene signatures (a typical study produces multiple results). Each signature was tagged with relevant ontology terms. This annotation step identified a total of 120 unique normal tissue concepts, 700 disease, 1,430 compound, and 135 genetic perturbation concepts (including gene mutations, knockouts, and siRNA knockdowns). The final dataset contained a high-dimensional space of gene signatures with ranked genes and associated ontology concepts (tags) for diseases, tissues, compounds, and genetic perturbations (Table 1).

A number of factors must be considered when performing a comparative analysis of highly heterogeneous data from different sources, platforms, and technologies. We applied statistical methods to compensate for differences in platforms and their probe content, in organisms studied, and in signature sizes that could arise from choices of analysis stringency (e.g. p-value cutoffs). In addition, directional information (up- or down-regulation) is important for assessing connectivity between different gene sets derived from gene expression data. Our rank-based directional enrichment analysis enabled us to statistically assess pairwise correlations between any two gene signatures and to use this information to rank connectivity between different biological states.

We applied our algorithm to compute pairwise correlation scores between all signatures in our system (Figure 2). The magnitude of the pairwise correlation score reflected the similarity of the two signatures, which is measured by the extent that the genes in one signature set are enriched at the top ranks of the other signature set, and vice versa. Each signature consisted of a list of genes that passed a select fold change, p-value, or other test statistic threshold. Thresholds might vary between different researchers, types of studies, and analytical methods, and thereby result in different signature sizes. To capture the key enrichment signal, even among very short or very long signatures, we developed a non-parametric rank-based statistical approach.

The general design of the algorithm, which we call “Running Fisher” (see Methods), is analogous to the Gene Set Enrichment Analysis (GSEA) method [11], [19]. As with GSEA, Running Fisher dynamically detects the most significant enrichment signal in a ranked signature, allowing the signature to contain a relatively more comprehensive collection of genes than would otherwise be required when using a stringent statistical cutoff. This “dynamic” enrichment detection approach overcomes the limitations of a more commonly used “selection” approach where a too stringent cutoff might lead to potential loss of significant information, and a too relaxed cutoff might include insignificant data into the evaluation [20]. The Running Fisher algorithm differs from GSEA in the assessment of the statistical significance, where p-values are computed by a Fisher's exact test rather than by permutations (see Methods for details). Overall, this approach provided us the flexibility to compute correlation scores for data of different sizes and filter thresholds, as well as the ability to use ranks in both query and target signatures.

The directional relationship between the two signatures was captured by the sign of the correlation score. The up-regulated genes and the down-regulated genes were separated into directional subsets, and correlation scores were computed for each directional subset from one signature against each subset from the other signature (Figure 3). A positive sign was given to a subset pair that changed expression in the same direction, and a negative sign was given to a subset pair that changed in opposite directions. The overall correlation score was the sum of directional subset scores, and the sign of the sum determined whether the two signatures were positively or negatively correlated (see Methods for details). Using this strategy, we computed pairwise correlation scores between all 25,000 signatures, resulting in over 625 million pairwise scores.

Currently, our system contains tens of thousands of datasets representing diverse types of biological conditions. With the development of new sequencing technologies we anticipate hundreds of thousands of public datasets to be available for the research community in the near future. To systematically interrogate these huge quantities of data, we have to abstract our analysis to the level of biological conditions those datasets represent. Researchers can then look at the connections between their own data and the potentially thousands of individual datasets with matching tissues, diseases, compounds, or genetic perturbations. Ontology-based meta-analysis is designed to accomplish that goal by computing an overall correlation score between a given gene set and an ontology concept (e.g. disease). The meta-analysis algorithm statistically assesses “reproducibility” of significant findings, thus minimizing the chance of random correlations and poor-quality data affecting the final results.

The meta-analysis algorithm aggregated scores for various ontology terms associated with the correlated signatures, weighted by the strength of the correlation score (Figure 4, Methods). This was computed separately for tissue, disease, compound, and genetic perturbation categories. The algorithm considered any available hierarchical relationships of ontology terms and propagated enrichment scores to more general concepts accordingly. Concepts that may have had lower scores than their parent concepts were clustered under the parent. A ranked structure of the most relevant tissues, diseases, and compounds was thus pre-computed for each signature. The advantage of this strategy is that related ontology tags could be associated with each signature semantically. For example, “heart” and “left heart ventricle” can both contribute to the “heart” concept meta-analysis score since “heart” is the parent concept of “heart ventricle”.

As each new signature was added, the meta-analysis computations for existing signatures in the system were also updated. This ongoing process ensured that at a given time the most up-to-date results of the meta-analysis given the current state of the knowledge base were produced. When a query with a given set of genes was performed a total collection of meta-categories, as well as individual signatures were scanned to identify top-ranking normal tissues, diseases, and compounds (Table 1).

Seale et al. identified new progenitor cells that gave rise to brown fat and muscle cells but not to white fat cells [22]. Currently, however, there is no gene expression data available for these new precursor cells. Given that, we decided to investigate the molecular properties of another brown fat cell precursor and its molecular similarity with normal tissues and cell types. We derived a brown preadipocytes signature consisting of 2,302 probesets (mouse MG_U74Av2 Affymetrix chip) by comparing gene expression of brown to white preadipocytes (GEO Accession # GSE7032, Table S4) [23]. Using this signature, we performed a correlation analysis across all normal tissues and cell types in NextBio (Figure 5B), and found that muscle stem cells were among the top five concepts with the strongest positive correlation scores to the brown preadipocytes signature (Table 3). This result further contrasts the differences between brown and white fat cells and demonstrates the association of muscle to brown fat. We continued our analysis by looking for patterns as brown and white adipocytes undergo differentiation, and we continued to observe a pattern of positive correlation between brown adipocyte and muscle cell precursors. In accordance with results in Seale et al., our meta-analysis approach suggests the possibility of the existence of a common precursor cell for these two cell types [22].

A majority of studies currently processed within NextBio adhere to certain criteria for inclusion:

To ensure standardization in the processing pipeline, studies were processed from raw data whenever available and from pre-processed data otherwise. For example, for Affymetrix-based studies in which CEL files are available, RMA normalization was applied [36]. Otherwise, expression summary intensities, such as those processed generated by MAS5 (Affymetrix) or dChip were processed [37]. All datasets went through appropriate processing steps that depended on the data type, platform, and experimental design used to generate the data and include:

The vast majority of processed data in the system falls into the category of case-control experimental design analyzed using Welch or standard t-tests, paired or unpaired, as appropriate. Quality assessment methods were employed to review sample-level and dataset-level integrity – these included curator review of pre- and post-normalization boxplots, missing value counts, and p-value histograms (after statistical testing) with FDR analysis to determine whether the number of significantly changing genes is greater than expected by chance.

A p-value significance cutoff of 0.05 (without any multiple testing correction) and a minimum absolute fold-change cutoff of 1.2 (typically the lowest sensitivity threshold of commercial microarray platforms) was used to obtain the final set of signatures of differentially-expressed genes. This double filtering procedure serves to address different aspects of variability in the data [38]–[40]. To address the potential unreliability of the fold-change metric at low intensity levels [41], genes with signals lower than a 20^th percentile cutoff in both control and test groups were discarded from the signature.

Expression profiles can vary considerably from study to study and from platform to platform. Different platform technologies can yield different dynamic ranges, distributions of fold-changes, and p-values that reflect the technologies used. To allow inter-study comparability, a non-parametric approach was established so that ranks were assigned to each final gene signature based on the magnitude of fold change. Fold-change, as a ranking metric, had a better concordance across platforms than p-values from statistical tests [42]. Ranks were then further normalized to eliminate any bias due to varying platform sizes.

In the absence of a “gold standard” for processing microarray data, these statistical threshold cutoffs serve to maintain a reasonable and consistent level of data quality across all studies analyzed within NextBio and are commonly adopted in the literature [43]–[45]. The thresholds are intentionally permissive to ensure that signatures contain all potentially interesting elements. The potential for introducing noise, i.e., more false positives, is balanced by (a) enforcing the basic quality control metrics described above and (b) incorporating a normalized rank-based scheme that captures the relative importance of each gene in a signature. This key metric of the meta-analysis framework is described below. In summary, this strategy results in the following advantages:

An index of microarray platforms was compiled to aid in the comparison of microarray data. The index provides a standardized mapping of commonly used public- and vendor-specific vendor gene identifiers to reference identifiers such as NCBI Entrez Gene, UniGene, Ensembl, RefSeq, or GenBank accession numbers.

To enable seamless comparison across different species, orthologs were identified for each pair of organisms and were grouped into ortholog clusters. Ortholog information was derived from Mouse Genome Informatics (MGI) at Jackson Lab (http://www.informatics.jax.org), HomoloGene at NCBI (http://www.ncbi.nlm.nih.gov), and Ensembl (http://www.ensembl.org). Ortholog clusters were generated as follows: 1) the manually curated pairwise ortholog data among human, mouse, and rat from MGI were retrieved and clustered to form initial ortholog clusters. 2) The homology group data among human, mouse, rat, fly, and worm were analyzed to remove those in conflict with MGI data. The filtered homology group data were then entered into the ortholog clusters. 3) The whole genome pairwise sequence similarity data from Ensembl were processed to identify reciprocal best hits as candidate orthologs for all pairwise organisms among human, mouse, rat, fly, worm, and yeast. The candidate orthologs were prioritized based on the percentage sequence identity and examined against the existing ortholog cluster. Qualified ortholog candidates were then entered into the ortholog cluster.

The directional relationship between the two signatures is captured by the sign of the correlation score. The up-regulated genes (b⁺) and the down-regulated genes (b⁻) are separated into directional subsets, and correlation scores are computed for each directional subset from one signature (b1⁺, b1⁻) against each subset from the other signature (b2⁺, b2⁻). A positive sign is given to a subset pair of the same direction (b1⁺b2⁺, b1⁻b2⁻), and a negative sign is given to a subset pair of opposite directions (b1⁺b2⁻, b1⁻b2⁺). The overall correlation score is the sum of directional subset scores and the sign of the sum determines whether the two signatures are positively or negatively correlated (Figure 3). The matching genes between two typical gene signatures are depicted in Figure 2. The directionality and ranks for each gene is shown.

The detailed steps given two gene signature sets (b1, b2) are as follows:

First, each gene signature set is rank-ordered according to fold change, p-value or a particular score. If appropriate metrics are not provided, then the gene signature set is unranked. The up-regulated genes and down regulated genes are noted with positive and negative signs to imply directionality, respectively. A directional subset is generated for each direction, such as b1⁺, b1⁻, b2⁺, and b2⁻ (Figure 3). If no directional data are provided, then the gene signature set is not directional and only one subset is formed with the whole signature set, such as b1^o, or b2^o.

Second, all the subset pairs are identified: b1Di, b2Dj, where Di and Dj are the available directions (+, −, or o) in b1 and b2, respectively. The Running Fisher algorithm is applied to each subset pair. The top ranking genes in the first subset b1Di are collected as a group G, and the second subset b2Dj is scanned top to bottom in the rank order to identify each rank with a gene matching a member in the group G. If the subset is unranked, all the genes in the subset are retrieved at the first scan.

At each matching rank K, the scanned portion of the second subset b2Dj consists of N genes, and the overlap between group G and N genes is M. A Fisher's exact test is performed at rank K, to evaluate the statistical significance of observing M overlaps between a set of size G and a set of size N, where the set of size G comes from platform P1 and the set of size N comes from platform P2, given the sizes of P1 and P2 as well as the overlap between P1 and P2.

At the end of the scan, the best p-value is retained, and a multiple hypothesis testing correction factor is applied. The multiple testing factor is the expected number of overlaps between the two subsets of the given sizes, given the two platforms P1 and P2. The negative log of the multiple testing corrected best p-value is a score for the subset pair.

Next, the Running Fisher algorithm is performed in the reverse direction: the top ranking genes in the second subset b2Dj are collected as a group G, and the first subset b1Di is scanned in the rank order. The same procedure in this reverse direction produces another score for the same subset pair. The two scores are averaged to represent the magnitude of the similarity between the two subsets. A positive sign is given to the final subset pair-score if Di and Dj are the same. A negative sign is given if Di and Dj are opposite. The score is unsigned if any of Di and Dj is not directional.

Finally, the overall score is computed by summing up all directional subset pair scores (Figure 3). The sign of the sum determines whether the two signatures are positively or negatively correlated. If one of the two signature sets is directional and the other is not directional, the overall score is represented by the larger of the two subset pair scores, annotated with the contributing direction from the directional signature. If both signatures are not directional, then a single unsigned pair score is calculated between the two biosets.

Given a gene signature representing the set of genes of interest from a given experiment, a meta-analysis of the tens of thousands of tagged gene signatures in the NextBio system can be used to determine tissues, diseases, and compounds associated strongly with the query set. Conceptually, the problem is that of ranking ontology terms (concepts) based on how strongly enriched signatures tagged with those concepts are with a set of genes of interest. For a set of genes, ranked or unranked, the gene set enrichment analysis described above is used to identify other strongly associated signatures. Based on the strength of the association, the aggregated scores for their associated semantic concepts were computed.

Meta-analysis scores were computed separately for tissue, disease, and compound categories. Given a query gene signature, a list of contributing signatures was obtained for each category based on two criteria – (1) They have enrichment scores with the query signature above a pre-determined threshold and (2) They pass an initial screening logic that ensures that they are tagged with the appropriate combination of concepts to allow them to contribute to that category.

Based on the list of contributing signatures, along with the associated concepts and enrichment scores, the equation below describes the various factors considered in determining a score for a concept.

The normalized hit count for a concept is the sum of the ratio of associated score of each signature tagged with that concept to the overall best association score. The background count of a concept is the number of signatures in the NextBio system tagged with that concept. Inclusion of the background count reduces the bias toward popular concepts that have more associated gene signatures than others. Finally, the average weighted rank represents the average rank of a tagged signature relative to all other correlated signatures weighted by the associated normalized score.

Given the dynamic nature of the NextBio system where the distribution of data from various species, platforms, data types, and semantic categories changes on a continuous basis, it is not obvious at the outset what the relative contributions of each of these factors should be toward determining an optimal overall scoring function for determining top ranked concepts. These are determined empirically by optimizing the model described above using gold standard use cases and tuning parameters (a,b,c in the equation above) for each of the factors.

This meta-analysis scoring process results in a ranked list of ontological terms for each tissue, disease, and compound category. It should be noted that some concepts are part of a hierarchical ontological framework. When that was the case, enrichment scores for signatures tagged with specialized concepts are accordingly propagated to more general parent concepts in the hierarchy. After scores for all concepts are computed, children concepts with lower scores than parent concepts are clustered and presented to the user.

The overall direction of association (positive or negative) between a query signature and a concept refers to the net correlation of the query signature with a set of contributing signatures (see previous section) tagged with that concept. Recall that these contributing signatures may have positive or negative pairwise correlation scores with the query signature. A cumulative positive and negative score is obtained by aggregating the scores for the positively and negatively associated signatures separately. To minimize the effect of any spuriously strong signature association, the cumulative scores are each weighted by the ratio of the respective number of positively or negatively associated signatures to the total number of contributing signatures. Finally, the overall direction is called depending on which weighted cumulative score is greater, as long as a minimum difference threshold is met.