There is a simple test for scale invariance that comes from the above analysis. To derive it, in the original x, y-coordinates and using the model equation y = αx, whatever regression procedure is used will result in the slope α depending on the data. This is written as α = α(x, y). Using the scaling X = x/S_x and Y = y/S_y, the model equation becomes Y = α ¯ X, where α ¯ = α S x / S y. Now, scale invariance requires that α ¯ = α ( X , Y ). Combining these two results, the requirement for scale invariance is that α ( x , y ) = S x S y α ( x / S x , y / S y ) , (4) for any positive values of S_x and S_y. Using the infinitesimal generators S_x = 1 + ε and S_y = 1 + δ, then the above equation takes the form [14] α ( x , y ) = 1 + ε 1 + δ α ( x / ( 1 + ε ) , y / ( 1 + δ ) ) = α ( x , y ) + ε [ α ( x , y ) - ∇ x α ( x , y ) ] - δ [ α ( x , y ) + ∇ y α ( x , y ) ] + ⋯ , where ∇_x is the gradient in the x variables (and similarly for ∇_y). The O(ε) and O(δ) requirements are that x · ∇ x α ( x , y ) = - α ( x , y ) , and y · ∇ y α ( x , y ) = α ( x , y ) .

These are easily solved using the n-dimensional version of spherical coordinates. Specifically, letting x = r X ( ϕ 1 , ϕ 2 , ⋯ , ϕ n - 1 ) , where r = ||x||₂ and the ϕ_i’s are the angular coordinates, then x ⋅ ∇_x α = −α reduces to r∂_r α = −α. The general solution of this is α = c/r, where c can depend on the ϕ_i’s. Doing something similar for y, the conclusion is that, to be scale invariant, the minimizer must depend on the data as α = | | y | | 2 | | x | | 2 F , (5) where F can depend on the values of the angular coordinates for x and y.

To determine the requirement for rotational invariance, assuming the angle of rotation is θ, then ( x ′ y ′ ) = ( cos θ sin θ - sin θ cos θ ) ( x y ) .

The model function now has the form y′ = α′ x′, and invariance requires that α ′ = α cos θ - sin θ α sin θ + cos θ .

Writing the minimizer of the error function as α = f(x, y), then α′ = f(x′, y′) results in the requirement that f ( x , y ) cos θ - sin θ f ( x , y ) sin θ + cos θ = f ( x cos θ + y sin θ , - x sin θ + y cos θ ) .

Using the infinitesimal generator θ = ε, then the O(ε) requirement is y · ∇ x f - x · ∇ y f + f 2 + 1 = 0 , (6) where f = f(x, y). As it should, the solution in Eqs (2) and (3) satisfies this nonlinear partial differential equation. What does not satisfy the equation is Eq (5). It is easiest to illustrate this assuming there are only two data points. In this case, the 2-dimensional spherical coordinates used in Eq (5) can be written as x₁ = r cos k, x₂ = r sin k, y₁ = R cos K, and y₂ = R sin K. Substituting Eq (5) into Eq (6), and reducing gives ( R 2 r 2 + 1 ) cos ( k - K ) F + sin ( k - K ) ( R 2 r 2 ∂ k F + ∂ K F ) = 1 + R 2 r 2 F 2 .

Given that F is independent of r and R, the above equation leads to the following two equations cos ( k - K ) F + sin ( k - K ) ∂ k F = F 2 , and cos ( k - K ) F + sin ( k - K ) ∂ K F = 1 .

These equations are solvable by introducing the change of variables s = k − K, t = k + K, and from this one finds that it is not possible to find a function F that satisfies both equations. In other words, there is no function which satisfies both Eqs (5) and (6).

The conclusion from the above analysis is that it is not possible for the minimizer to be both scale and rotationally invariant. It is known that there are rotation and reflectively invariant error functions, and an example is one that uses true distance (E_D in Fig 1). So, the question considered here is whether there are error functions which satisfy all of the stated conditions except for rotational invariance.

It is relatively easy to find scale invariant error functions. For example, using the usual (vertical) least squares error E y ( α ) = ∑ i = 1 n ( α x i - y i ) 2 = α 2 x · x - 2 α x · y + y · y , (7) the minimum occurs when α = x ⋅ y/x ⋅ x. Similarly, using the (horizontal) least squares error E x ( α ) = ∑ i = 1 n ( x i - y i / α ) 2 = x · x - 2 x · y / α + y · y / α 2 , (8) the minimum occurs when α = y ⋅ y/x ⋅ y. Both of these are scale invariant. What E_x and E_y are not, however, are reflectively invariant. To be reflectively invariant it is required that irrespective of which variables are considered independent or dependent, that an equivalent result is obtained. This means that the minimizer of E_x is the same as the one obtained for E_y. Mathematically, the requirement is that α ( y , x ) = 1 α ( x , y ) . (9)

The error function to be considered here is based on the geometric mean of the ordinary least squares error functions. In the case of two variables, the error function is E ( α ) = E x ( α ) E y ( α ) , (10) where E_x and E_y are given in Eqs (7) and (8), respectively. Minimizing this one finds that α ( x , y ) = ± y · y x · x , (11) where the + is used if x ⋅ y > 0 and the − is used if x ⋅ y < 0. This satisfies the change of variables condition Eq (4), and this guarantees α is scale invariant. It is also reflectively invariant because it satisfies Eq (9). Another way to conclude that it is reflectively invariant is to note that the error function Eq (10) is a symmetric function of E_x and E_y.

The minimizer in Eq (11) is well-known and can be obtained in a number of ways. This includes using geometric mean regression [9], using the geometric means of the minimizers for Eqs (7) and (8), and by using simple symmetry arguments [10]. What is not known is a way to generalize it to multilinear regression to obtain scale and reflectively invariant low dimensional approximations of data. What is presented below is one way this might be possible.

For ordinary least squares, one of the standard measures on how well the linear model fits the data is the coefficient of determination R². For the multidimensional generalizations of Eq (10) considered later, a natural measure of fit involves the centroid of the error simplex. To explain what this is for this two dimensional problem, and connect it with R, let α_x and α_y be the minimizers of E_x and E_y, respectively. The centroid in this case is α_c = (α_x + α_y)/2. The error measure to be introduced concerns how α, the minimizer of E, differs from the centroid, relative to the width of the simplex. The resulting formula is C F = | α - α c α y - α x | .

Now, using the law of cosines x ⋅ y = ||x||₂ ⋅ ||y||₂cos θ, where θ is the angle between x and y, then α_x = α/cos θ and α_y = α cos θ. Moreover, R² = cos² θ, which gives a (signed) value of R = cos θ. Combining these formulas, the result is C F = 1 2 · 1 - | R | 1 + | R | .

Consequently, the better the fit (the closer R² is to one), the closer the minimizer of E is to the centroid of the error simplex formed from the minimizers of E_x and E_y.

As an example, in R 3, for the line with α₂ = 3 and α₃ = 5, 40 randomized points within a distance of 0.2 of the line are used for the data (see Fig 2). For notational simplicity, let p₁ = x, p₂ = y, p₃ = z, α₂ = α, and α₃ = β. The location of the minimizer of E was found using MATLAB’s fminsearch command, and the location is shown in Fig 3. Also shown are the locations of the minimizers for E_x, E_y, and E_z, determined by Eqs (14) and (15), as well as the associated triangular region formed by these three points. For comparison, the location of the solution as determined using an unscaled PCA is shown. An important observation coming from this figure is that the minimizer for the scale (and reflectively) invariant error function E = (E_xE_yE_z)^1/3 is located near the centroid of the triangle. This is expected because of Eq (17).

It is worth having a way to characterize how close the minimizer and centroid are, to provide a measure for the goodness of fit. One possibility is to use the maximum distance relative to the width of the simplex, as given by the formula C F = max { | α - α c | w α , | β - β c | w β } , (18) where (α_c, β_c) is the centroid, w_α = max{α_x, α_y, α_z} − min{α_x, α_y, α_z}, w_β = max{β_x, β_y, β_z} − min{β_x, β_y, β_z}. In these expressions, (α_x, β_x), (α_y, β_y), and (α_z, β_z) are the minimizers of E_x, E_y, and E_z, respectively, and they determine the error simplex in Fig 3. For the solution shown in Fig 3, C_F ≈ 0.01.

Finally, in finding the minimizer the question of whether or not the error function is convex arises. To verify this, its contour and surface plot are shown in Fig 4. Note that, because of the assumed form of the model function in this example, the error functions E_x, E_y, and E_z, and E are undefined when either α or β are zero. The statement that E is convex refers to its dependence on α and β in the quadrant in which the minimizers are located.

As before, for notational simplicity, let p₁ = x, p₂ = y, p₃ = z, α₂ = α, and α₃ = β. The model function can then be written as z = αx + βy. In this case, from Eq (19), the individual error functions are: E x = ∑ i = 1 n ( x i - z i / α + β y i / α ) 2 = x · x - 2 x · z / α + 2 β x · y / α + z · z / α 2 - 2 β y · z / α 2 + β 2 y · y / α 2 , (28) E y = ∑ i = 1 n ( y i - z i / β + α x i / β ) 2 = y · y - 2 y · z / β + 2 α x · y / β + z · z / β 2 - 2 α x · z / β 2 + α 2 x · x / β 2 , (29) E z = ∑ i = 1 n ( z i - α x i - β y i ) 2 = z · z - 2 α x · z - 2 β y · z + α 2 x · x + 2 α β x · y + β 2 y · y . (30)

The minimizer α_x of E_x is α x = 1 ( x · y ) ( y · z ) - ( x · z ) ( y · y ) ( y · z - y · y - x · z x · y )( y · z z · z ) , (31) the minimizer α_y of E_y is α y = 1 ( x · x ) ( y · z ) - ( x · z ) ( x · y ) ( y · z - x · y - x · z x · x )( x · z z · z ) , (32) and the minimizer α_z of E_z is α z = 1 ( x · x ) ( y · y ) - ( x · y ) 2 ( y · y - x · y - x · y x · x )( x · z y · z ) . (33)

The resulting error function based on the geometric mean is E ( α , β ) = ( E x E y E z ) 1 / 3 = 1 α 2 / 3 β 2 / 3 ( z · z - 2 α x · z - 2 β y · z + α 2 x · x + 2 α β x · y + β 2 y · y ) . (34)

Taking the first partials of this, and setting them to zero, one obtains the (nonlinear) system α 2 x · x + α β x · y + β y · z = z · z (35) β 2 y · y + α β x · y + α x · z = z · z . (36)

These equations can be written in somewhat simpler terms by letting q = α x · x z · z and p = β y · y z · z .

Using the law of cosines, then Eqs (35) and (36) become q 2 + p q cos θ x y + p cos θ y z = 1 (37) p 2 + p q cos θ x y + q cos θ x z = 1 , (38) where θ_xy is the angle between x and y (with similar definitions for the other angles). An analytical formula for the solution of these equations is not apparent, but it is a simple matter to compute the solution numerically.

As an example, for the plane with α = −10 and β = 6, 40 randomized points within a distance of 0.15 of the plane were used for the data. The location of the minimizer of E was found by solving Eqs (35) and (36) using MATLAB’s fminsearch command, and the location is shown in Fig 5. Also shown are the locations of the minimizers for E_x, E_y, and E_z, and the associated triangular region formed by these three points. For comparison, the location of the solution as determined using an unscaled PCA is shown. As expected from Eq (27), the of E minimizer is located near the centroid of the triangle. To quantify the difference, using the formula in Eq (18), C_F ≈ 0.02. Finally, to demonstrate the convexity of the error function in the octant containing the minimizer, the contour and surface are shown in Fig 6.

As an illustration, and a comparison with a PCA, consider the data for the seven major crime rates and population for the larger cities in the U.S in 2009 [15]. Of the 105 cities reported, 79 were randomly chosen for the training set, and the testing dataset consists of those left out. With this, n = 79 and m = 8. The minimizer of Eq (20) was found using a modified Polak-Ribière descent procedure, with Armijo’s method used to solve the line search problem. The modification is that if the search direction determined using Polak-Ribière is not a direction of descent, then the steepest descent direction is used instead. A description of the Polak-Ribière and Armijo’s methods can be found in [5, 16]. The starting point for the descent procedure was a convex combination of the minimizers for the E_j’s, which was computed using the formulas given earlier. Also, the normalization for the PCA uses | | p j | | 2 / n, for each column j of the centered training data matrix.

The resulting testing versus training comparison for each variable is shown in Fig 7. For Eq (20), the α_j’s are determined by minimizing E and then using those same values for each graph. For example, for the graph associated with p₁, the model function is rewritten as p 1 = - α 2 α 1 p 2 - α 3 α 1 p 3 - ⋯ - α m - 1 α 1 p m - 1 + 1 α 1 p m , (39) and the resulting training-testing values are plotted. For the PCA, a seven component approximation is made. To make a more quantitative comparison between these two approaches, and because both methods are reflectively invariant, the normalized least squares errors for each of these graphs is given in Table 1. The normalization used is N||p_j||₁/n, where n is the number of values in the training set and N the number in the testing set. It is seen, at least in this example, that the values using Eq (20) produce a uniformly better result than those obtained using a PCA. To make the point that the fit using a PCA varies with the scaling, the errors using two other scalings are also given in Table 1. The scaling used for PCA₁ is considerably worse than the PCA values, while the PCA₂ values in the last column are distinctly better.

As a second example, data mining has been used to predict tasting preferences for wine [17]. The dataset for red wine consists of 1599 instances (vectors) containing values for 12 attributes, 11 being physicochemical properties of the wine and one the quality score for taste. The physicochemical properties in this case are: fixed acidity (α₁), volatile acidity (α₂), citric acid (α₃), residual sugar (α₄), chlorides (α₅), free sulfur dioxide (α₆), total sulfur dioxide (α₇), density (α₈), pH (α₉), sulphates (α₁₀), and alcohol (α₁₁). Following the protocol used in [17], the training set consists of 2/3 of the original (randomly chosen), and the testing set the remaining 1/3. They also used a regression error characteristic (REC) curve, which is defined in [18], to evaluate how well various regression procedures predict the taste score. To compare how Eq (20) does, using the training set, the minimizer is computed with the modified Polak-Ribière method described earlier, treating the data as defining a smooth function. The resulting REC curve determined using Eq (20) is shown in Fig 8. Also shown are the curves obtained using a PCA as well as from using a standard multivariable least squares regression. The dashed curve is what is obtained using Eq (20) if the predicted values are converted back into integer scores as originally used in [17]. Finally, for the two and three dimensional examples considered earlier, it was found that the centroid of the error simplex furnished a fairly accurate approximation for the minimizer. A measure of how close they are is given in Eq (18). Generalizing this formula for the wine example, it is found that C_F ≈ 0.15, which indicates they are reasonable close.

It is evident from Fig 8 that standard least squares provides better predictive values for the taste score than when using Eq (20). This can be quantified using the mean absolute deviation (MAD), which is defined as ||y − y*||₁/N, where y* are the test values, y are the predicted values, and N is the number of observations in the testing set (note that the values are unscaled). For Eq (20) the MAD value is 1.5, while using least squares it is 0.5. Consequently, if given a particular bottle of red wine, least squares would provide a better predictor of how it tastes. What Eq (20) provides is a better model for how to modify the physicochemical properties to achieve a particular taste score, and the reason is reflective invariance. To illustrate, with the coefficients computed previously, the resulting MAD values for the other possible training-testing cases are given in Table 2. The fact that the least squares values are so poor is not surprising, and the reason was given earlier. Namely, the values obtained using p_m as the dependent variable are not equivalent to the values obtained if the model equation is solved for, say, p₁, and then considering it as the dependent variable.

Wave height at Buoy Station 46006, located in the northern Pacific Ocean, is measured hourly, along with the wind direction, wind speed, wind gust, dominant wave period, average wave period, barometric pressure, and water temperature [19]. The specific data considered here come from measurements over approximately 11 months. The dataset consists of 7960 instances containing values for the eight attributes. A reduced version of this dataset was examined in [20], in comparing how various regression procedures do on real ecological data. The reduction was to use the daily average values rather than the hourly data, but all of the values are considered here. Following the protocol used in [20], the training set consists of 3/4 of the original (randomly chosen), and the testing set the remaining 1/4. The comparison was made using the mean squared prediction error (MSPE), which is defined as | | y - y * | | 2 2 / n, where y* are the test values for the wave height, y are the predicted values, and n is the number of observations in the testing set (note that the values are unscaled). Using Eq (20), the MSPE is about 1.2, which compares favorability with the mean MSPE of 12.3 found in [20]. The resulting MAD value is about 0.8, and the REC curve is shown in Fig 9. Finally, for this example, C_F ≈ 0.16. This indicates that the centroid of the error simplex and the minimizer are close, although not as close as in the earlier examples.

The chemical oscillator known as the Belousov-Zhabotinskii reaction can be described with the following five reactions [21] A + Y → X + P , X + Y → 2 P , A + X → 2 X + 2 Z , 2 X → A + P , Z → 1 2 f Y .

The chemicals here are bromous acid (X), bromide (Y), cerium-4 (Z), bromate (A), and a product P. Given known initial concentrations for each species, measurements are made at later times to follow the evolution of the overall reaction. The complication is that one or more of these reactions are very fast, or occur at very low concentrations, so accurate measurements are difficult. What is demonstrated here is how the hyperplane fit can be used in the case of when four of the five concentrations are measured, and the fifth is determined using regression. What is important is that no matter which four are chosen, that the same (equivalent) value is obtained for the fifth species. To demonstrate, the reactions were run for 20 different initial concentrations, and the values for the five species were recorded at 60 second intervals up to 10 minutes. The resulting dataset consists of 180 instances, and these were randomly split into a training set (3/4) and a testing set (1/4). The values fitted are those for Z. Using Eq (20), the MAD value is 0.015 and the MSPE is 4.6 × 10⁻⁴. Using a standard least squares fit the values are 0.012 and 3 × 10⁻⁴, respectively. However, if you fit the values for, say, A, and then transform back to the equation for Z, then the MAD and MSPE values using Eq (20) remain unchanged while the least squares values are 0.024 and 10⁻³. As a final comment, using a reflectively invariant error function with chemical density fits has the distinct advantage of being able to determine possible conservation laws inherent in the system.

The link between phytoplankton and water chemistry has been the subject of several recent studies, although the connections with specific chemical species are incompletely understood. Several correlations have been made, and the physicochemical parameters most commonly considered are oxygen, pH, NH₄-N (ammonium nitrogen), NO₃-N (nitrate nitrogen), and PO₄-P (phosphate phosphorus) [20, 22–24]. The latter study used the values for the Chlorophyll-a density, along with various chemical properties, of lakes in the Northeast that were measured over a four year period [25]. Altogether, there are 20 useable variables in this study, and 500 observations. After removing incomplete entries and others that are not useable, the data set consists of 348 observations. The model requires specification of which variables to use, and after some analysis of the data it comes down to five: total dissolved aluminum, nitrate, total nitrogen, total phosphorous, total suspended solids, and turbidity. In comparison, in [20], 10 variables were initially used. Using Eq (20), the resulting MSPE is about 1.1, which compares favorability with the mean MSPE of 2.4 found in [20]. The resulting normalized MAD value is about 0.7, and the REC curve is shown in Fig 10. Finally, for this example, C_F ≈ 0.13. This indicates that the centroid of the error simplex and the minimizer are close, similar to what was found for the wave height example.