Let X = { x 1 , . . . , x n 1 } and Y = { y 1 , . . . , y n 2 } two sets whose elements are in R p, and m t = ( m 1 , m 2 , . . . , m n 1 ) and w t = ( w 1 , w 2 , . . . , w n 2 ) the vectors whose components are the weights of the elements of X and Y respectively, positive and such that ∑ i = 1 n 1 m i = ∑ j = 1 n 2 w j = 1.

Weights can be assigned depending on the importance of the individual in the sample. This could be of interest if the individuals are collectives; for example: cities or universities; that can be weighted by their size. If all the individuals are equally important, weights must be m_i = 1/n₁∀i and w_j = 1/n₂∀j.

Definition 1. A hyperplane H in R p is an (p-1)-affine set and can be represented as H = { x ∈ R p | a t x = b }, where b ∈ R and a ∈ R p, a ≠ 0_p, and they are unique up to a common non-zero multiple.

Initially, we look for a hyperplane H that strictly separates X from Y (Fig 1). If such hyperplane does not exist, we focus on a hyperplane that minimizes a measure of the deviation of this goal, called separation error.

Proposition 1. X and Y are strictly separable if and only if there exists a hyperplane H = { x ∈ R n | a t x = b } such that the following system: σ = { a t x i ≥ b + 1 , i = 1 , … , n 1 a t y j ≤ b - 1 , j = 1 , … , n 2 (1) is consistent. Such hyperplane is named separator hyperplane.

Proof.

If X and Y can be strictly separated, there exists c ∈ R p, c ≠ 0_p and d ∈ R such that: c t x i > d , i = 1 , . . . , n 1 c t y j < d , j = 1 , . . . , n 2 (2)

Let ε_i and δ_j the slacks of each constraint in (2): ε i = c t x i - d > 0 , i = 1 , . . . , n 1 δ j = d - c t y j > 0 , j = 1 , . . . , n 2 and η ≔ min{ε_i, δ_j}. We can define ( a t , b ) = η - 1 ( c t , d ) , (3) where a ∈ R p, a ≠ 0_p and b ∈ R.

Multiplying both sides of (3) by ( x i t , - 1 ) t we have: a t x i - b = η - 1 ( c t x i - d ) = η - 1 ε i ≥ 1 , i = 1 , … , n 1

Similarly, multiplying (3) by ( y j t , - 1 ) t we have: a t y j - b = η - 1 ( c t y j - d ) = - η - 1 δ j ≤ - 1 , j = 1 , … , n 2

Therefore, the pair (a, b) is a solution of the system (1).

Conversely, if the system (1) has a solution (a, b), then H = { x ∈ R p | a t x = b } verifies {a t x i ≥ b + 1 > b , i = 1 , … , n 1 a t y j ≤ b - 1 < b , j = 1 , … , n 2

Moreover, a ≠ 0_p (otherwise, b + 1 ≤ 0 ≤ −1). Hence, H is a hyperplane separating strictly X and Y.

Such hyperplane will be referred to as a separator hyperplane. This proposition leads us to locate sets X and Y as it is showed in Fig 2, regarding the hyperplanes H = { x ∈ R p | a t x = b } , H + 1 = { x ∈ R p | a t x = b + 1 } and H - 1 = { x ∈ R p | a t x = b - 1 } .

On the other hand, if (1) is an inconsistient system, there exists some i ∈ {1, …, n₁} for which b + 1 − a^t x_i > 0 or j ∈ {1, …, n₂} for which a^t y_j − b + 1 > 0. Therefore, we can take the following values as error measure of each element: max { b + 1 - a t x i , 0 } , i = 1 , … , n 1 max { a t y j - b + 1 , 0 } , j = 1 , … , n 2 .

Adding all these measures weighted by m_i and w_j, respectively, we obtain the function f, called separation error function: f ( a , b ) : = ∑ i = 1 n 1 m imax { b + 1 - a t x i , 0 } + ∑ j = 1 n 2 w jmax { - b + 1 + a t y j , 0 } . (4)

The separation error function is a non-negative, convex and non-differenciable function and the aim is to solve the problem ( P 1 ) minf ( a , b ) .

Proposition 2. σ is consistent if and only if v(P₁) = 0. In such case, any optimal solution of P₁ defines a separator hyperplane.

Proof.

If X and Y can be strictly separated, there exists ( a ¯ , b ¯ ) solution of σ. So, f ( a ¯ , b ¯ ) = 0 and ( a ¯ , b ¯ ) is an optimal solution of (P₁).

Conversely, if ( a ¯ , b ¯ ) is an optimal solution of (P₁), each term in f will be equal to zero and by Proposition 1, X and Y can be separated strictly. Moreover, a ¯ ≠ 0 p because f ( 0 p , b ¯ ) = 2 and it can not be an optimal solution of (P₁).

So, X and Y can be strictly separated if and only if v(P₁) = 0. But, in any case, the objective is translated in finding the solution ( a ¯ , b ¯ ) to the problem (P₁). We approach this task through a linear problem equivalent to (P₁), whose optimal solutions will define our discriminant hyperplane, that is, the hyperplane that minimizes the separation error function.

Proposition 3. (P₁) is equivalent to the problem ( P ) M i n ∑ i = 1 n 1 m i u i + ∑ j = 1 n 2 w j v j s . t . u i ≥ - a t x i + b + 1 , i = 1 , … , n 1 u i ≥ 0 i = 1 , … , n 1 v j ≥ a t y j - b + 1 , j = 1 , … , n 2 v j ≥ 0 j = 1 , … , n 2 where the objective is finding (a, b), that define the hyperplane, with the support of the variables u_i and v_j that identify potential errors to be minimized.

Proof.

Since each of the functions to maximize in each operand in (4) is convex, we have max { b + 1 - a t x i , 0 } = min { u i ∈ R + ∣ u i ≥ b + 1 - a t x i } and max { a t y j - b + 1 , 0 } = min { v j ∈ R + ∣ v j ≥ a t y j - b + 1 }

It allows us to reformulate our initial problem as the equivalent problem (P) in the following sense [13]:

(P) is a solvable problem and every optimal solution will provide a discriminant hyperplane that minimizes the separation error as long as a ≠ 0_p. We can state that if v(P) = 0, this situation is guaranteed by Proposition 2 but, if v(P) > 0, we need to add a very weak condition on the data sets (in the sense that it will usually be verified), what we will prove in the following proposition. Let us remember that in a linear problem, a necessary and sufficient condition for x ¯ to be an optimal solution is that the objective vector can be written as a non-negative linear combination of the active constraints on x ¯.

Proposition 4 If v(P) > 0 and x ¯ ≠ y ¯, with x ¯ = ∑ i = 1 n 1 m i x i and y ¯ = ∑ j = 1 n 2 w j y j there exist an optimal solution of (P) that gives a discriminant hyperplane.

Proof.

Let us suppose that ( a ¯ , b ¯ , u ¯ , v ¯ ) is an optimal solution of (P) with a ¯ = 0 p. Then, u i ≥ b + 1 , for all i = 1 , … , n 1 (5) v j ≥ 1 - b , for all j = 1 , … , n 2 (6) and all of them will be active (otherwise, the solution is no an optimal solution). So we can consider u ¯ = ( b ¯ + 1 ) 1 n 1 and v ¯ = ( 1 - b ¯ ) 1 n 2, where 1 n 1 and 1 n 2 are vectors with all its elements equal to 1 in R n 1 and R n 2, respectively. Then, v ( P ) = ∑ i = 1 n 1 m i ( b ¯ + 1 ) + ∑ j = 1 n 2 w j ( 1 - b ¯ ) = 2 .

We will distinguish the different values of b in order to determine the active constraints in each case and apply the condition that characterizes the optimality.

In the case of more than two groups, we could proceed in two different ways:

In lpda package we have implemented the second option.

In nowadays it is very usual being involved in projects where the number of measured variables is much higher than the number of samples. In such cases, the high dimension allows statistical methods were succesfull separating groups. However, the hyperplane can overfit the training data and as a result a bad evaluation in the data test is obtained. To avoid this problem we propose obtaining the hyperplane from Principal Components (PCs) instead of the original variables. In general, when managing large amounts of noisy but correlated data, data analysis can greatly benefit from the application of dimensionality reduction methods, such as PCA, which allows the identification of the main patterns of variability avoiding residual or non-structural variation (examples in [14, 15]). Such approaches are effective in providing global understanding of most relevant information that can help to detect the differences between the studied groups.

PCA reduces the dimension of a set of individuals measured in a p-dimensional basis, taking advantage of the relationship between the variables. The method consists of projecting the individuals on a subspace of dimension q < p extracting the major information. The solution of this problem is the subspace defined by the q eigenvectors associated with the q higher eigenvalues of the variance-covariance matrix of the data. The selected number of PCs, q, is typically obtained on the basis of the percentage of the explained variability or by a cross-validation criterion. The PCA model corresponding to a data matrix X, of dimensions n × p, gives us the following decomposition: X = 1 n μ t + T P t + E (9) where 1_n is a size n column vector of ones, μ^t is a size p row vector containing estimates of de average for each variable, scores of the individuals in each PC are collected in the matrix T, the loadings (eigenvectors) are given by the matrix P and the residuals are collected in E.

The aplication of LPDA to the scores, or T matrix, will provide a classification hyperplane that avoids the undesirable noise focussing in the signal of interest. For more details about the PCA model and other projection techniques see [16].

SVM is a hyperplane-based classification method, as said in the introduction. This method tries to find the hyperplane with the maximum margin that separates two classes, allowing some errors in the training set to avoid overfitting [4, 5]. Although SVM can also perform a non-linear classification, when dealing with so many variables there is no need of additional flexibility that will give polynomial or radial kernel models. For this reason, in next section we use linear classifiers, also called Support Vector Classifiers, for RNA-Seq example.

From the different packages avaible in R to apply SVM [19] we use the SVM implementation called e1071. The needed parameters in each application were computed with the crossvalidation proccess available in this package.

Logistic Regression considers a linear model where the response, a binary variable representing the class, is modelled with a logistic transformation. It is considered a specific case of Generalised Linear Models that are a generalization of classical Linear Models, which can accommodate a wider class of distributions named as exponential family, providing great flexibility for modeling different types of response variables. Normal, Poisson, Binomial and Gamma are examples of this family of distributions. In Logistic Regression, Binomial distribution is considered to model the response. More details in [3].

LDA computes the probabilities of belonging to each of the groups according to the available variables using Bayes Theorem (posteriori probability) and Normal distribution. The predicted class will be the one whose posteriori probability is maximum [1, 2].

Poisson Discriminant Analysis (PDA) [20] and Negative Binomial Discriminant Analysis (NBDA) [21] are specific methods for RNA-Seq samples classification. They can be considered as an extension of the LDA because they are Bayes rule-based classifiers taking into account the discrete count distribution inherent in these data.

As SVM and LPDA are methods based in hyperplanes separation, it is worth taking a closer look at this comparison. By comparing results of SVM and LPDA to different data sets we have seen that working with a high number of variables or having clear differences between groups, both methods are succesfull separating groups. However when having few variables or existing overlaps between groups, we find some differences. As example we show pairwise variables comparison of palmdates concentration data. We consider 4 variables: fibre, fructose, sorbitol and myo-inositol, avoiding glucose because it is highly correlated with fructose and gives repeated results. Fig 3 shows cases where both methods are successfull but the hyperplanes are slight different and Fig 4 shows cases where LPDA gets less separation errors than SVM: only one predicted error with LPDA in the third comparison meanwhile there are 1, 2 and 7 errors respectively with SVM.

Another advantage we have found in LPDA with respect other techniques in several data sets is the good treatment of unbalanced data. These data is frequently encountered in biomedical and bioinformatics studies, where the group it is desired to predict is much smaller than the other one. General methods try to minimice the global error thus disadvantaging the minority class. Specific techniques are emerging for dealing with this problem [22]. Weights m_i and w_j considered by LPDA inside each group, mitigate this problem, meanwhile other techiques as SVM need to specify additional arguments when this situation arrises [23].

Default data is an example of unbalanced data that illustrates the problem clearly because only 0.3% of the data belongs to the group of interest (default class) that is desired to predict with low error. Table 1 shows sensitivity, especificity and the clasification error obtained with LPDA, weighted-SVM, Logistic Regression and LDA. We call weighted-SVM results of SVM applied considering as weights for each class the inverse of their sizes. As the interest is the good prediction in the default class, identified as the positive class, we must focuss in the sensitivity or percentage of True Positives detected. We observe that LDA and Logistic Regression give low sentitivity meanwhile LPDA gives a sensitivity very near the obtained with weighted-SVM and higher specificity, thus less global error.

We applied LPDA, SVM, POlda and NBlda, to the cervical cancer data described before. Firstly, all the data was considered to compute the number of classification errors as a training set. None error was detected with LPDA and SVM. However, POlda and NBlda gave 3 and 4 classification error respectively, therefore, LPDA and SVM give a separate hiperplane meanwhile methods based in distributional assumptions do not.

We also evaluated the methods in test sets with a cross-validation strategy where the model was obtained 1000 times in different training and test sets. Table 2 shows the classification error rates average jointly to their confidence intervals. First, we notice the importance of the dimension reduction (LPDA-PCA) in this case, and in general when dealing with high dimensional data as RNA-Seq, which significantly reduces the error rate. We also observe that LPDA-PCA results are very similar to SVM and NBlda meanwhile LPDA without PCA results are similar to the POlda approach.