To assess the theoretical adequacy of accuracy, we generated some samples of the space of joint count distributions for input and output classes and instances of classification with a prescribed accuracy (see Section sec:datasets for the details). Then, their entropy decomposition was calculated and plotted in the ET (see Section sec:mms, sec:entropy-triangle). Figure 2 presents the cases with , with and with .

A number of observations can be gleaned from this figure:

We are driven to conclude that accuracy is not a trustworthy criterion to judge the degree to which a particular classification process transfers information from the input class distribution to the output decision class distribution.

The question poses itself whether it is possible to conjoin accuracy and mutual information transfer in a single measure. To provide an affirmative answer to this we first state the hypothesis:

Consider a confusion matrix for a classifier obtained from instances of classification pairs. The lowest accuracy is that of a classifier returning a uniform count matrix: the most balanced testing dataset will distribute to each class and a clueless classifier will further redistribute these uniformly to each output class as instances. Since the diagonal has cells, the diagonal sum is

It is bounded by and any value smaller than the lower bound is an sure indication that a permutation of the output tags will ensure higher classification accuracy, that is, a better mapping of input to output class names.

Consider the perplexity reduction chain of Fig. 3. To the extent that the number of input classes and their distribution is a given—whereas is a construct of the classifier—we want to concentrate on measuring how well the input class distribution was learned by the training process, that is, in the prior class distribution perplexity reduction of equation (1). Regarding the classifier training algorithm, is a given and cannot be modified, whereas quantifies the amount of successfully learned information. More importantly for our purposes, is the amount of information the classifier failed to learn. Therefore the EMA appears naturally as a quality measure based in the remaining perplexity of the variable(3)

Since is the entropy of ignored by the classifier, as per our hypothesis and in the absence of any other source of information this is the expected performance of the classifier with equivalent (possibly fractional), equally likely classes: the higher this number, the worse the classifier will be.

To illustrate this, notice that when the training process of the classifier has been able to capitalize on all mutual information to leave no remaining perplexity, , whence . Similarly, , either because the classifier has utterly failed to capture any information between and , , or because the entropy of the data was minimal, .

Notice that when the entropy of is not maximal then whence and the EMA detects an artificial lower bound for (2), . The artifice here is that this increase does not depend on the training of the classifier but on the prior class distribution. This suggests including a correction into equation (3) to account for the deviation from uniformity in the prior class distribution so that(4)with when both and , implying that and . Note that if and only if . Unlike the case of , the eventuality that the data are not uniformly distributed is corrected on , as entails . Moreover, the further away from a uniform prior class distribution to the classifier, the worse its upper range bound will be. Eventually, for —which implies by equation (2) whence —we have, again, the worst possible value of the measure, . Notice that in this accuracy-optimal case , but in an unhelpful way. Essentially, making the input data less random impacts the ability of the classifier to capitalize in mutual information to bind together input and output, and this is registered by the measure. The normalized information transfer factor can be rewritten as,

(5)Note also that NIT factor does not depend directly on the input or output distribution. Conveniently, since the normalized information transfer factor is a monotonic function of normalized mutual information the relative height in the ET offers a visual tool to quickly inspect such effectiveness. Finally, when evaluating a set of systems in the same task, is constant throughout the evaluation, so , and they offer the same ranking results, easily visualized in the ET.

For the reasons above, we posit the EMA in (3) to measure the performance of classification tasks, and the NIT factor in (4) or (5) to measure the effectiveness of the classifier learning process.

In this Section we present an example of how to use the EMA and the NIT factor in automatic classifier evaluation tasks. We consider the case of the MEG mind reading challenge organized by the PASCAL (Pattern Analysis, Statistical modeling and ComputAtional Learning) network [23]. Since accuracy was the “official” evaluation criterion, for comparison purposes Fig. 4.fig: (A) presents the results in the entropy triangle ordered by accuracy as reflected in the coloring of the points. System at was deemed the winner with close behind at . In a detail of the dense region of harder competition in Fig. 4.(B) clusters , and are evident. We next suggest a procedure to analyze the classification performance of a population of classifiers:

The result of this process is an assessment of a (population of) classifiers, whereby one may discuss the advantages of EMA and NIT factor vis–vis other performance measures, for instance, accuracy. Further examples of using this procedure to evaluate classification tasks can be found in the File S1.

Perplexity has already been used as a performance measurement for language modeling where it refers to the expected average of alternatives a model has at every word history [24]. It is also often used as an off-line method for speech recognition task evaluation following the intuition that a classifier using a lower-perplexity model will outperform a higher-perplexity one, all other things equal.

It cannot be stressed enough that since the EMA and the NIT factor concentrate in the prior class distribution and mutual information, it is harder for classifiers to boost their performance by manipulating the posterior class distribution through specialization: only the increase in information transfer through will improve the evaluation figure.

Considering robustness, the EMA, being a harsher, worst-case criterion, might be more deserving of trust than easygoing and unreliable accuracy to, for instance, guide decision making. It certainly has a more interpretable and less easily bendable criterion—specially if reporting the classification error is not the ultimate goal. Furthermore, in cases where —for instance, when using a “reject” class—the EMA and the NIT factor are still defined, whereas accuracy is problematic, and not very much used.

As seen in the MEG mind reading example, the EMA and the NIT factor are capable of determining whether a task has been effectively solved or not. But it cannot distinguish whether this is caused by technical limitations in the classifier selection process or because the task is inherently “hard”. Only the kind of iterated classification effort of community research that attempts many different classifier-building techniques on the same task can be effective for this purpose.

Nevertheless, the effective input perplexity can ensure that, methodologically at least, the task is “as hard as it should be” at . Furthermore, our developments show clearly that a failure to maintain prior class distribution uniformity in the design or capture of the task data entails that the expected mutual information—therefore the NIT factor —captured by any possible classifier that solves the task can never reach maximal levels. This is a strong guideline for prospective collectors of datasets, although data balancing strategies after data collection can also be used to achieve this goal [19].

Several other measures have sprouted to deal with the inadequacies of accuracy such as the Area-Under-the-(ROC)-Curve [8], [25], the Variation of Information [21], the Relative Classifier Information [26], the Confusion Entropy [10], [27] or Cohen's Kappa [13], but their use is not widespread, specially for the non-binary case, due to complexity of calculation, disparate purposes or each measures' own shortcomings. For instance, the AUC first needs to find a (multiclass) ROC representation of the task by obtaining multiple classifiers, possibly with the help of a parameter in the classifier learning process. The trading for good-vs-wrong decisions in terms of the parameter can then be judged from the Area-Under-the-ROC curve, which is then a measure on the learning method or model. In contrast, EMA would provide a different point in the ET for each classifier whence the best of these classifiers could be chosen. Complementarily, on the population of classifiers, a statistical description of the NIT factor could be used to assess the learning capabilities of the method.

In classification proper, to illustrate the disparity of the conclusions that can be reached with alternative performance measures, we have included in File S1 a comparison of the classical Matthew Correlation Coefficient (MCC) [28] and the Confusion Entropy (CEN) [27]—whose similarities are also explored in [10]–on three different classifications tasks: the MEG Mind Reading task already explored, the TASS sentiment analysis task [29]–both machine learning tasks—and the well-known Miller & Nicely human perceptual capability exploration task [5].

For each task we provide the heat maps of the confusion matrices (Figs. S1, S2 and S4 in File S1) as customary. We also provide the tables detailing perplexities, EMA, NIT factor, and MCC' related values (Tables S1, S2 and S3 in File S1 ). The entries in the tables are ordered by accuracy. For the TASS and M&N data we also supply the ET's with the color bar according to EMA, and MCC' (Figs. S3 and S5). Their comparison, detailed in the File S1 Section, reveals that MCC' is highly correlated with accuracy in ranking results and shows similar shortcomings. Even though CEN performs a little better, it is highly biased towards majority classifiers providing over optimistic assessment for them. Notably, once the ET, EMA and the NIT factor have shed light on the problem, reassessment of prior evidences for either CEN or MCC prove them not to be so advantageous in evaluating classifiers.

Consider two discrete random variables and and their joint probability distribution . An entropy diagram somewhat more complete than what is normally used for the relations between their entropies was presented in [12] and is here depicted in Fig. 5(A). We distinguish in it the familiar decomposition of the joint entropy as the two entropies and whose intersection is . But notice that the increment between and is yet again , hence the expected mutual information appears twice in the diagram. Further, the interior of the outer rectangle represents —with and the uniform distribution on inputs and outputs—,the interior of the inner rectangle , and is their difference. Finally, the variation of information was found to be an important quantity in [21]. Putting together this information results in the balance equation for information related to a joint distribution,which can be further normalized in ,(6)and represented in a De Finetti or ternary diagram as the equation of the -simplex in normalized space, hence the name entropy triangle, ET.

The position of the coordinates of a classifier on the Entropy Triangle characterizes its performance, and we use this characterization to visually assess it indicated in Fig. 6. Classifiers at the apex or close to it obtain the highest accuracy possible on balanced datasets and transmit a lot of mutual information, hence they are the best classifiers possible. Those at the left vertex or close to it are dealing with balanced data but doing a bad job of utilizing it: they are the worst classifiers. Those at the right vertex or close to it are dealing with very easy, unbalanced data and claiming very high accuracy, yet they are not learning anything from it: they are specialized (majority) classifiers and our intuition is that they are the kind of classifiers that generate the accuracy paradox [16].

Notice that in equation (6), since both and and and are independent as marginals of and , respectively, we may write:what suggests writing separate balance equations for each variable,

The formulae above and the occurrence of twice the expected mutual information in equation (6) suggests a different information diagram, depicted in Fig. 5(b): both variables and now appear somehow decoupled—in the sense that the areas representing them are disjoint—yet there is a strong coupling in that the expected mutual information appears in both and . It is important to note that both decompositions can be represented in the same (split) entropy triangle as equation (6) dictates. The technique is explained in [12].

The space of square confusion matrices, of sizes and a given number of input samples, , depicted in Fig. 2 was obtained by first generating every possible partition of with parts as input distributions , allocating input samples in each of the input classes. In this way, the set of all possible input class distributions, from uniform to the most skewed , is obtained. Then, for each of the previous distributions, every possible weak composition of with parts is produced, yielding sets of all the possible distributions for each of the rows of . Finally, the Cartesian product of those sets produces every possible combination of rows corresponding to the selection of one element in every one of the sets. Except from row permutations —that would only amount to a reordering of the input classes— this procedure guarantees the presence of every possible .

The MEG mind reading task aims at decoding the identity of a video stimulus based on magnetoencephalography (MEG) recordings done during naturalistic stimulation [23]. In particular, subjects were exposed to video stimuli of different classes: a first category of short clips (6–26 s. long) with being artificial stimuli (screen savers showing animated shapes or text), being natural stimuli (sceneries like mountains or oceans) and being football stimuli (from —European— football matches) and a second category of long clips (approximately 10 min. long) with being television series (from “Mr. Bean” in particular) and being films (from Chaplin's “Modern times”). The goal was to classify unlabeled test examples into these classes based on the MEG signal alone. The competition took place in March, 2011 and 10 participants submitted their classifiers whose confusion matrices are analyzed in this paper. The data was provided upon request from the organizers of the competition.

The MATLAB(A registered trademark of The MathWorks, Inc.) code to draw the entropy triangles in Figures 2 and 4 has been made available at: http://www.mathworks.com/matlabcentral/fileexchange/30914