Criterion (7) is invariant under positive rescaling of the input matrices C_k, thus the AJD matrix B of {a₁C₁,…,a_KC_K} is the same as the AJD matrix of {C₁,…,C_K} for any positive set {a₁,…,a_K}.

The proof is trivial using well-known properties of the determinant and of the logarithm and will be omitted here. For AJD solutions according to a given criterion, we make use of the following:

The AJD of set {C₁,…,C_K} according to some AJD criterion is well defined (or is essentially unique) if any two global minimizers B₁ and B₂ of the criterion are in relation B₁ = PΔB₂, where P is a permutation matrix and Δ is an invertible diagonal matrix.

For details on the essential uniqueness of AJD see [40, 45, 46]. This is a generic property in the sense that in the presence of noise the AJD is essentially unique (with very high probability). Now, consider again data model (1) and the theoretical generative model of SOS statistics of the observed data as per (2), where A is the (unknown) mixing matrix and the D_k, with k∈{1,…,K}, are the (unknown) diagonal matrices holding K SOS statistics of the source processes. Working with AJD we are interested in the situation where the noise term in (1) is small enough so that the AJD solution is well defined as per definition 1. We will then make use of a general property of such well-defined AJD solutions:

For any invertible matrix F, if B is an essentially unique AJD of set {FC₁F^T,…,FC_KF^T}, then ΔPBF is the AJD of set {C₁,…,C_K}, where invertible diagonal matrix Δ and permutation matrix P are the usual AJD scaling and permutation indeterminacy, respectively.

Saying that B is an essentially unique AJD of set {FC₁F^T,…,FC_KF^T} according to some AJD criterion implies that the set {BFC₁F^TB^T,…, BFC_KF^TB^T} is a global minimizer of the AJD criterion employed. Thus, matrix BF is, out of possible different scaling and permutation as per definition 1, a global minimizer of the same AJD criterion for the set {C₁,…,C_K}.

Finally, we will need the following:

Let B, with inverse A, be an essentially unique AJD of set {C₁,…,C_K}; an AJD criterion is said to verify the self-duality invariance if PΔA^T is a well defined AJD of set {C₁^-1,…,C_K^-1} satisfying the same criterion, with invertible diagonal matrix Δ and permutation matrix P the usual AJD scaling and permutation indeterminacy, respectively.

The Fisher information metric allows us to measure the length of curves in M and find the shortest curve between two points Ω and Φ on M. This is named the geodesic and is given by [2] γβ(Ω→Φ)=Ω12(Ω−12ΦΩ−12)βΩ12, 8 where β is the arc length parameter. When β = 0 we are at Ω, when β = 1 we are at Φ and when β = 1/2 we are at the geometric mean of the two points (Fig 2).

The exponential and logarithmic maps are shown graphically in Fig 2. The function that maps a vector V∈T_ΩM to the point Φ∈M following the geodesic starting at Ω, is named the exponential map and denoted by Φ = Emap_Ω(V). It is defined as

The inverse operation is the function mapping the geodesic relying Ω to Φ back to the tangent vector V∈T_ΩM. It is named the logarithmic map and denoted V = Lmap_Ω(Φ). It is defined as

Given two points C₁ and C₂ on the manifold M, their Riemannian distance based on the Fisher information metric is the length of the geodesic (8) connecting them. It is given by [3, 10, 16] δR(C1↔C2)=‖ln(C2−1C1)‖F=trln2(Λ)=∑nln2λn, 11 where Λ is the diagonal matrix holding the eigenvalues λ₁,…,λ_N of either matrix (4) or (5). This distance has a remarkable number of properties, some of which are reported in Table 1 [14, 15, 25]. For more inequalities see [2, 25]. Associated to the chosen metric is also the Riemannian norm, defined as the Riemannian distance from an SPD matrix C to the identity, that is, the Euclidean distance of its logarithm to the zero point:

The Riemannian norm is zero only for the identity matrix, while the Frobenius norm is zero only for the null matrix. Either an eigenvalue smaller or greater than 1 increases the norm and the norm goes to infinity as any eigenvalues go to either infinity or zero. Importantly, because of the square of the log, an eigenvalueλ increases the norm as much as an eigenvalue 1/ λ does (see Fig 3), from which the invariance under inversion.

Given a set of SPD matrices {C₁,…,C_K}, in analogy with the arithmetic mean of random variables, a straightforward definition of the matrix arithmetic mean is A{C1,⋯,CK}=1K∑kCk, 13 and a straightforward definition of the harmonic mean is

On the other hand a straightforward definition of the geometric mean is far from obvious because the matrices in the set in general do not commute in multiplication. Researchers have postulated a number of desirable properties a mean should possess. Ten such properties are known in the literature as the ALM properties, from the seminal paper in [11]. The concept of Fréchet means and the ensuing variational approach are very useful in this context: in a univariate context the arithmetic mean minimizes the sum of the squared Euclidean distances to K given values while the geometric mean minimizes the sum of the squared hyperbolic distances to K given (positive) values. In analogy, we define the (least-squares) Riemannian geometric mean G{C₁,…,C_k} of K SPD matrices C_k such as the matrix satisfying [13, 15]. argminG{C1,⋯,CK}∑kδ2(G{C1,⋯,CK}↔Ck), 14 where δ²(⋅↔⋅) is an appropriate squared distance. In words, the Riemannian geometric mean is the matrix minimizing the sum of the squared distances of all elements of the set to itself. Using the Fisher information (FI) distance (11) such mean, which we name the FI mean and denote as GR{C1,⋯,CK} or, shortly, GR{Ck}, features all ALM properties. The FI mean is the unique SPD geometric mean satisfying non-linear matrix equation [15] ∑ k ln( G R −1 { C 1 ,⋯, C K } C k )=0 15 or, equivalently, ∑kln(GR−12{C1,⋯,CK}CkGR−12{C1,⋯,CK})=0, 16 where, in line with the notation used in this article, Gs{⋅} stands short for (G{⋅})s, for any power s. We have listed a number of properties of the FI mean along with some related inequalities in Table 2 (see also [2, 25, 47, 48]). Note that in the literature this least-squares geometric mean is sometimes referred to as the barycenter, the Riemannian center of mass, the Fréchet mean, the Cartan mean or the Karcher mean (although these definitions in general are not equivalent and attention must be paid to the specific context, see e.g., [31]).

Given two points C₁ and C₂ on the manifold M, the Geometric Mean of them, indicated in the literature by C₁#C₂, has several equivalent closed-form expressions, such as [2, 12, 25, 28, 47] C1#C2=C112(C1−12C2C1−12)12C112=C1(C1−1C2)12=(C2C1−1)12C1 17 And

In the above the indices 1 and 2 can be switched to obtain as many more expressions. The geometric mean of two SPD matrices is indeed the midpoint of the geodesic in (8) and turns out to be the unique solution of a quadratic Ricatti equation [2, 48], yielding

Our investigation has started with the following:

The FI geometric mean of two SPD matrices can be expressed uniquely in terms of their joint diagonalizer; let B be the JD (3) of matrices {C₁, C₂} such that BC₁B^T = D₁ and BC₂B^T = D₂ and let A = B⁻¹, then the geometric mean is (C1#C2)=A(D1D2)12AT, 19 for any JD solution, regardless the permutation and scaling ambiguity.

Since diagonal matrices commute in multiplication, using (3) and properties (50) and (54) of the geometric mean we can write B(C₁#C₂)B^T = D₁#D₂ = (D₁D₂)^½. Conjugating both sides by A = B⁻¹ we obtain (19).

A consequence of the scaling indeterminacy of the JD and (19) is that we can always chose B such that BC1BTBC2BT=I, 20 in which case A is a square root ([49], p. 205–211) of the geometric mean, i.e.,

Given a set {C₁,…,C_K} = {C_k} of K>2 SPD matrices the point G_R{C_k} satisfying (23)–(24) has no closed form solution. The research of algorithms for estimating the FI geometric mean or a reasonable approximation for the case K>2 is currently a very active field [3, 24, 28–30, 48, 50]. In this work we consider two iterative algorithms for estimating the FI mean. The first is the aforementioned gradient descent algorithm [10]. In order to estimateG_R{C_k}, we initialize M with the arithmetic mean or any other smart guess. Then we apply the iterations given by M←M12exp[υK∑kln(M−12CkM−12)]M12, 22 until convergence. Here υ is the step size, typically fixed at 1.0. Notice that this algorithm iteratively maps the points in the tangent space through the current estimation of the mean (10), computes the arithmetic mean in the tangent space (where the arithmetic mean makes sense) and maps back the updated mean estimation on the manifold (9), until convergence (Fig 4). Note also that the global minimum attained by gradient descent (22) satisfies ∑kln(M−12CkM−12)=0, 23 upon which the iterate does not move anymore. We have already reported that these iterations have linear convergence rate, but convergence itself is not guaranteed. In order to minimize the occurrence of divergence and also to speed-up convergence, while avoiding computationally expensive searches of the optimal step size, we use in the present work the following version with heuristical decrease of the step-size over iterations:

Recently it has been proposed to use the least-squares (Fréchet) geometric mean (14) based on the log-Euclidean distance [21, 22, 51] and the Bhattacharyya divergence [23, 24], also named S-divergence [25, 26], which turns out to be a specific instance of the α-divergence setting α = 0 [27]. These studies suggest that these alternative definitions of geometric mean give results similar to the FI mean in practical problems, while their computational complexity is lower.

The Log-Euclidean distance is δL(C1↔C2)=‖ln(C1)−ln(C2)‖F, 25 which is the straightforward generalization to matrices of the scalar hyperbolic distance and, from property (48), equals the FI distance if C₁ and C₂ commute in multiplication. The interest of this distance is that the geometric (Fréchet) mean (14) in this case has closed-form solution given by

This is a direct generalization of the geometric mean of positive scalars and, again, from property (48), it equals the FI mean if all matrices in the set pair-wise commute. The computation of (26) requires fewer flops than a single iteration of the gradient descent algorithm (22), so this constitutes by far the most efficient geometric mean estimation we know. The log-Euclidean mean (26) possesses the following properties: invariance by reordering, self-duality, joint homogeneity and determinant identity [21–22]. It does not possess the congruence invariance, however it is invariant by rotations (congruence by an orthogonal matrix). Also, important for the ensuing derivations, whereas the determinant of the log-Euclidean mean is the same as the determinant of the FI mean (since both verify the determinant identity), the trace of the log-Euclidean mean is larger than the trace of the FI mean, unless all matrices in the set pair-wise commute, in which case as we have seen, the two means coincide, hence they have the same trace and determinant [21]. Because of this trace-increasing property, the log-Euclidean mean is in general farther away from the FI mean as compared to competitors such as constructive geometric means [28]. Moreover, in general the log-Euclidean mean differs from the FI mean even for the case K = 2.

In [24] the author has shown that the Bhattacharyya divergence (also named log-det divergence and S-divergence) between two SPD matrices C₁,C₂∈ℜ^N·N behaves similarly to the FI distance if the matrices are close to each other. It is symmetric as a distance, however, it does not verify the triangle inequality. This divergence is δB2(C1↔C2)=ln|(C1+C2)/2|12ln|C1C2|=ln|A{C1,C2}||G{C1,C2}|=ln∏n12(λn+1λn), 27 where the arithmetic mean A{C1,C2} is defined in(21), the geometric mean G{C1,C2}=C1#C2 in(32)–(33) and the λ_n are, again, the N eigenvalues of either matrix (4) or (5). Successively, in [26] the author has shown that the square-root of the Bhattacharyya divergence is a distance metric satisfying the triangle inequality. Both the Bhattacharyya divergence and distance are invariant under inversion and under congruence transformation [24, 26]. The geometric mean (14) of a set of matrices based on divergence (27) is the solution to nonlinear matrix equation [26]

In order to estimate GB{C1,⋯,CK} we initialize M and apply iterations [24] M←K[∑k(Ck+M2)−1]−1, 29 which global minimum has been shown to satisfy (28) [26]. This mean possesses the following invariance properties: invariance by reordering, congruence invariance, self-duality and joint homogeneity. However, it does not satisfy the determinant identity. As a consequence, in general both the determinant and the trace of the Bhattacharyya mean differ from those of the FI mean. However, the Bhattacharyya mean of K = 2 matrices coincides with the FI mean [27].

Let H be an invertible matrix of which the inverse is G. Using the congruence invariance of the geometric mean (50) and conjugating both sides by G we obtain

Our idea is to reach an approximation of GR{C1,⋯,CK} by approximating GR{HC1HT,⋯,HCKHT} in the expression above. Particularly, if the matrices HC_kH^T are nearly diagonal, then they nearly commute in multiplication, hence we can employ property (54) to approximate the geometric mean GR{C1,⋯,CK} by expression GA''=AGL{BC1BT,⋯,BCKBT}AT=Aexp(1K∑kln(BCkBT))AT, 30 where GL{BC1BT,⋯,BCKBT} is the log-Euclidean mean introduced in (41), B is a well-defined AJD matrix of the set {C₁,…,C_K} (definition 1) chosen so as to minimize criterion (7), and A is its inverse. Because of the scaling (Δ) and permutation (P) ambiguities of AJD solutions, (30) actually defines an infinite family of admissible means. Before we start to study the specific instance of the family we are interested in (the closest to the FI mean in general), let us observe that if all products BC_kB^T are exactly diagonal then the family of means defined by (30) collapses on a single point in the manifold, which is indeed the FI geometric mean, for any AJD matrix with form PΔB. This is the case when the data are generated according to model (1) and the noise term therein is null, in which case the coincidence of the two means is a consequence of property (54) or, regardless the data model, if K = 2, in which case (30) reduces to (19). The same is true also whenever 1K∑kln(BCkBT)=0, 31 in which case the matrix exponential of (30) is the identity matrix and the family (30) collapses to the unique point AA^T just as for the case K = 2 in (21). Interestingly, the family of means (30) includes the log-Euclidean mean, in that setting A = B⁻¹ = I we obtain (26), but also the FI mean, in the sense that setting A = M^1/2 and B = A⁻¹ = M^−1/2 we obtain the global minimum of the FI mean (23). This is the point on the manifold we sought to approximate by approximating condition (31). First of all, we show that the AJD permutation ambiguity is not of concern here. We have the following

The family of means given by (30) is invariant with respect to the AJD permutation indeterminacy P, for any invertible AJD solution B with inverse A and any invertible diagonal scaling matrix Δ.

Taking into account the P and Δ indeterminacies, since for any permutation matrix P^T = P^-1 and for any diagonal matrix Δ = Δ^T, the family of solutions (30) has full form

If f (S) is a function of the eigenvalues of S (see section “Notation and Nomenclature”) and P is an arbitrary permutation matrix, then P^-1f (PSP^-1) P = f (S), thus, the family of solutions (30) actually takes form GA'{C1,⋯,CK}=AΔ−1exp(1K∑kln(ΔBCkBTΔ))Δ−1AT 32 for any invertible B = A^-1 and any invertible diagonal Δ.

Then, note that the family of means defined by (32) is SPD, which is a consequence of the fact that the exponential function of a symmetric matrix is always SPD and that both B and Δ are full rank. It also verifies the invariance under reordering (49), which is inherited from the invariance under reordering of AJD algorithms and of the log-Euclidean mean. Furthermore, we have the following

The family of means defined by (32) verifies the determinant identity(53), for any invertible matrix B = A^-1 and any invertible diagonal Δ.

We need to prove that for any invertible matrix B = A^-1 and any invertible scaling matrix Δ

Using (32) we write the left-hand side of the above equation such as |A||Δ−1||exp(1K∑kln(ΔBCkBTΔ))||Δ−1||AT|. Since the log-Euclidean mean possesses the determinant identity, this is

Developing the products of determinants and since |A|=|A−1|−1=|B|−1, we obtain the desired result such as

If B is the AJD solution of (7), with inverse A, the family of means defined by (32) verifies the joint homogeneity property (52), for any invertible diagonal Δ.

We need to prove that

The result follows immediately from the invariance under rescaling of criterion (7) (Proposition 1), as

So far we have considered the properties of the whole family of means with general form, (32) for which we have solved the permutation AJD indeterminacy (Proposition 4). We now seek the member of the family better approximating the FI geometric mean given a matrix PΔB that performs as the AJD of the set {C₁,…,C_K}. This involves choosing a specific scaling Δ. As we have seen, if in (32) exp(1K∑kln(ΔBCkBTΔ))=I, 33 then (21) is true also for K>2, in which case the FI geometric mean is given by AA^T, where A is the inverse of ΔB in (32) and (30) is a stationary point of (22). Such condition is well approximated if the left-hand side of (33) is nearly diagonal and all its diagonal elements are equal, that is, in our case, by scaling the rows of B such that

The uniqueness of this solution, regardless the choice of α, is demonstrated by the following

Let B = A^-1 be an invertible AJD of the set {C₁,…,C_K}. Scaling B so as to satisfy (34), the mean (32) is unique and invariant with respect to α.

Once matrix B satisfies (34), let us consider a further fixed scaling such us Δ = ωI, with ω>0. Substituting this fixed scaling matrix in the family of means (32) we have Aω−1Iexp(1K∑kln(ωIBCkBTIω))Iω−1AT, but this is Aω−1(ω2K)1Kω−1exp(1K∑kln(BCkBT))AT=Aexp(1K∑kln(BCkBT))AT, showing that the resulting mean does not change. Thus, given an AJD matrix B, we obtain our sought unique member of the family as GA{C1,⋯,CK}=Aexp(1K∑kln(BCkBT))AT,given(34)true for anyα>0, 35 where A is the inverse of the matrix B scaled so as to satisfy (34).

Without loss of generality, hereafter we will choose α = 1. With this choice the AJD matrix B is an approximate “whitening” matrix for the sought mean, since we have

In order to satisfy (34), notice that any change in Δ changes the log-Euclidean mean therein since the log-Euclidean mean is not invariant under congruent transformations, thus there is no closed-form solution to match condition (34) given an arbitray AJD matrix solution. However, we easily obtain the desired scaling of B by means of an iterative procedure (see below). We name the resulting approximation to the geometric mean (35) the ALE mean, where ALE is the acronym of AJD-based log-Euclidean Mean. The algorithm is as follows:

The ALE mean (35) satisfies the invariance under congruent transformation.

We need to show that for any invertible matrix F

Let B₁ a well defined AJD of set {C₁,…,C_K} with inverse A₁ and B₂ a well-defined AJD of set {FC₁F^T,…, FC_KF^T} with inverse A₂, both satisfying condition (35) for their respective set. The expression above is then

Because of Proposition 2, if matrix B₁ approximately diagonalizes the set {C₁,…,C_K} so does matrix B₂F according to the same criterion and if they both satisfy (35) for the set {C₁,…,C_K} they are equal out of a permutation indeterminacy that we can ignore because of proposition 4. As a consequence A₁ = (B₂F)^-1 = F^-1A₂ and thus A₂ = FA₁. Making the substitutions we obtain

The ALE mean verifies the self-duality property (51) if the AJD solution B, with inverse A, verifies the self-duality property of Definition 2.

Self-duality of the ALE mean is verified if

Using definition 2 we have GA−1{C1−1,⋯,CK−1}=[BTexp(1K∑kln(ATCk−1A))B]−1, and computing the inverse of the right-hand side

Using ln(S^-1) = -ln(S) and then (exp(-S))^-1 = exp(S) we have

Note that the AJD matrix satisfying criterion (7) verifies the self-duality of definition 2 only if K = 2 or if all products B^TC_kB are exactly diagonal, otherwise it verifies it only approximately. Therefore, in general our ALE mean estimation (35) verifies self-duality (51) only approximately. Interestingly, the ALE mean would verify the self-duality property as well if the AJD cost function makes use of a diagonality criterion based on the Riemannian distance (11) instead of the Kullback-Leibler divergence as in (7). However, the search of such an AJD algorithm has proven elusive so far. In conclusion, we have shown that the ALE mean verifies several important properties satisfied by the FI Riemannian mean (invariance under reordering and under congruence transformation, joint homogeneity and determinant equality). The self-duality property is satisfied approximately in general and exactly in some special cases. Next we will study the performance of the ALE mean by means of simulations.