In this study, three real datasets (Tab. 1, Fig. 1) were used either directly or to generate simulated data. They are unpublished datasets that will be used here just for the methodological purposes of this study while the analyses reflecting their biological relevance will be published elsewhere (P. Franchini, C. Fruciano et al. unpublished data). For all datasets, (semi)landmarks were digitized using tpsDig2 [24]. Landmark and semilandmark configurations were then aligned [25]–[26] in tpsRelw [27]. In all cases, the allometric component of shape variation was removed by performing, in MorphoJ [19], a multivariate regression of shape on centroid size. Residual Procrustes coordinates were then partitioned in two modules for the analyses of modularity (Tab. 1, Fig. 1). When performing Klingenberg’s method [16], significant modularity was found for all the datasets and subsets in Tab. 1 except for parental species and F1s in Dataset1 and for the CD subset of Dataset2, probably as a consequence of the small sample size of these subsets. All the subsequent analyses and simulations were performed in MATLAB (MathWorks, inc.).

To test if there is a relationship between the RV coefficient and sample size in real datasets, we performed a rarefaction procedure on the 308 configurations of Procrustes residuals belonging to F2 generation individuals in Dataset1. Briefly, we carried out the following procedure:: 1. we randomly sampled with replacement from the 308 configurations 100 samples for each of the sample sizes comprised between 10 and 300; 2. for each sample we computed the RV coefficient; 3. we calculated then the mean and standard deviation of the RV coefficient for each set of 100 samples at each sample size. We repeated the procedure described above 200 times and obtained grand means of the RV coefficients and average standard deviation by averaging the results of the 200 independent rarefaction analyses. We also computed the variance of the mean RV estimate across the 200 independent analyses. To assess the impact of different alignment procedures, we performed the above mentioned analysis both aligning all the 308 observations prior to the rarefaction procedure with a single generalized Procrustes analysis and performing separate generalized Procrustes analyses for each of the two modules. Further, to assess the effect of superimposition at each step, we carried out a simplified version of the procedure described above (steps 1–3). In this simplified procedure we performed a generalized Procrustes analysis (both performing a single alignment for all the landmarks and using separate alignments for each module) for each of the samples obtained at step 1 of the procedure for all the sample sizes comprised between 10 and 300 at steps of 5 (10, 15, 20,…).

Here we suggest that using a rarefaction procedure could be useful to overcome the problem of dependency of Escoufier RV on sample size when one needs to compare RV values directly or use them in downstream analyses. We apply this idea to the main groups (A. astorquii, A. zaliosus, F1 specimens, F2 specimens) of Dataset1. In particular, we draw – sampling with replacement –1000 random samples of 11 observations (sample size of the smallest group, F1 individuals) from each of the groups, compute the Escoufier RV coefficient for each dataset and, finally, compute a mean RV coefficient for each group.

Here we propose to test for the difference in RV between two a priori defined groups of observations.

Let A and B be two matrices n x t and m x t of, respectively, n and m observations and t variables (constituting, effectively, two a priori defined groups of n and m observations). Let t = q+r where q and r are two a priori defined groups of variables (representing modules). We propose computing the difference between RV coefficients for each group, defined as:

Where RV_A and RV_B are, respectively, the RV coefficient between q and r variables for matrix A and the RV coefficient between the same two sets of variables for matrix B.

We propose – to test the null hypothesis – creating two new groups A_PERM (of size n x t) and B_PERM (of size m x t) resampling without replacement observations from the pooled sample M of size (n+m) x t and computing the difference in RV between the two new groups as:

Where RV_APERM and RV_BPERM are the RV coefficients computed on A_PERM and B_PERM, respectively. After repeating the procedure N_PERM times to obtain an empirical distribution of RV_PERMDIFF, we will consider the proportion of times in which RV_OBSDIFF exceeds the differences of the empirical distribution as the probability level at which the null hypothesis H₀ of no difference in RV (modularity) between the two groups can be rejected.

We used this permutation test on Dataset2, comparing the RV coefficients for mice from the CD and ACR groups, and on Dataset3 comparing mice with standard and hybrid karyotype (in all cases 1000 random permutations were used).

To analyze the type I error for the permutation test proposed here, we used two different approaches: simulating no difference in modularity using real datasets and simulating no difference in modularity using random data.

For the first approach, we independently used the following sets of landmark configurations: F2 individuals from Dataset1, CD mice mandibles from Dataset2 and ACR mice mandibles from Dataset2. For each of the sets of landmark configurations independently and for each of the sample sizes comprised between 40 and 200 at steps of five, we generated 1000 random multivariate normal datasets with the observed mean and covariance. Then we subdivided each dataset in two subsets of equal number of observations (rounded to the next integer) and performed the proposed permutation test for the difference in RV coefficient. We, finally, used the proportion of significant (at the 5% probability level) tests at each sample size as a measure of type I error.

For the second approach, to test type I error in different conditions (different number of total cases, different number of total variables, different number of variables for each block, different number of observation for each a priori defined group), we generated 200,000 random datasets. We used random datasets as this choice allowed us to have a wide range in the number of variables and their possible subdivision in two blocks. Each dataset had a random number of observations comprised between 40 and 200 and a random number of variables comprised between 20 and 100. Observations for each variable were drawn from the standard uniform distribution on the open interval (0,1). Each random dataset was, further, divided into two groups of observations and two blocks of variables with the number of observations per group being random but constrained to be higher than 20 and the number of variables per block random but constrained to be at least four. For each of the simulated datasets we performed the proposed permutation test for the difference in the RV coefficient, using the proportion of significant tests as a measure of type I error. To investigate if the type I error of our test was dependent on number of cases, number of variables, difference in sample size between groups, difference in variables between blocks, we also produced plots of such features for datasets which returned a significant test.

In the case of a relatively small number of discrete samples with multiple observations (sampling sites), variation in geographic space could be studied by standardizing to a common sample size through rarefaction and/or multiple pairwise comparisons using the permutation test proposed here. However, in case of sparse sampling in geographic space, we suggest that a solution could be to compute, for each x_i observation in geographic space, the RV coefficient for the subset comprising x_i itself and its k-nearest neighbors, computing also the average spatial coordinates of the subset. In this way, for each of the original observations an RV coefficient and a new spatial position will be computed. The new matrix representing the variation in the RV coefficient in geographic space could then be analyzed with exploratory or hypothesis-testing tools. For instance, plots of the variation of the RV coefficient in geographic space can be obtained and functions to describe the observed patterns can be fitted to the data. Here we provide an example of such an approach using Dataset1. We divided a hypothetical bidimensional square – representing a hypothetical geographic space – into four quadrants. Then for three of the four quadrants we generated – using the multivariate normal distribution having the mean and covariance matrix of the F2 individuals in Dataset1–100 observations per quadrant and assigned them random uniform coordinates within the quadrant. For the fourth quadrant we generated 100 observations from the multivariate normal distribution having the mean and covariance matrix of a subset of F2 individuals chosen so to have a higher RV coefficient and assigned them random uniform coordinates. In such a way, we simulated the situation where within one quadrant – the upper left – there would be a maximum in the RV coefficient. We then computed the RV coefficient with the k-nearest neighbor approach outlined above using k = 20, Euclidean distances in the two-dimensional space as distance measure and the KD-tree technique for finding the k-nearest neighbors [28]. Finally, we fitted a third-order polynomial on the obtained matrix of RV coefficients and average spatial coordinates and we plotted the fitted surface as a surface plot and as a heat map.

Fig. 2 shows the average RV coefficient at each sample size obtained through rarefaction analysis. A clear pattern of decrease in RV at increasing sample sizes can be observed. Such pattern is especially pronounced at sample sizes smaller than 100, thus suggesting that comparing the RV coefficient across samples with different sizes can be misleading. It is interesting to notice that lower overall values and lower variances of estimates of the RV coefficient are obtained when performing separate generalized Procrustes analyses for each module (Fig. 2a,b). On the other hand, performing generalized Procrustes analyses for each dataset obtained resampling from the population does not affect the general finding of a decrease in the RV coefficient at increasing sample sizes (Fig. 2). The variance of the estimated average RV coefficient, as expected, decreases at increasing sample sizes (Fig. S1) and, while a critical sample size cannot be identified, the decrease is very pronounced at lower sample sizes.

In our example dataset, the average RV obtained after a rarefaction analysis at the smallest sample size (n = 11) for A. astorquii was 0.704, for A. zaliosus 0.623, for the F1 specimens 0.668, for the F2 specimens 0.554. These results might suggest that modularity in A. astorquii x A. zaliosus hybrids behaves as a transgressive trait. However, here the dataset has been just used as an example of using rarefaction to standardize the RV coefficient and any biological consideration will be made elsewhere.

The RV coefficients of mice belonging to the groups CD and ACR from Dataset2 were, respectively, 0.31 and 0.20 (difference 0.11; sample sizes were, respectively, 18 and 66). However, the newly developed permutation procedure did not reject (p = 0.758, based on 1000 permutations) the null hypothesis of equal RV coefficients. On the other hand, using the same test on standard and hybrid mice from Dataset3 (RV coefficient 0.31 and 0.20, respectively, with sample sizes 45 and 41) resulted in significant differences (p = 0.02). The fact that in the two datasets, the same difference in the RV coefficient produced non-significant and significant results, respectively, suggests that the observed difference in RV between the two groups in Dataset2 can be explained as a consequence of their difference in sample size.

In the analysis of type I error using mean and covariance from real datasets, the average type I error for all simulation was 0.049. Plots of type I error at different sample sizes (Fig. 3) do not show any relationship between type I error rate and sample size.

In the analysis of type I error using random datasets with variable number of cases, number of observations, number of observations in each group and number of variables in each block, the overall type I error was 0.05. Plotting the number of cases, number of observations, fraction of observations in the first group, fractions of variables in the first block, difference between number of variables and number of cases for occurrences of type I error showed no departure from random expectations (Figure S2). This shows that type I error for the suggested permutation test is not influenced by the analyzed features.

In our example analysis using randomly distributed observations, a third-order polynomial provided a reasonably good fit (R² = 0.69) for modeling the variation of the RV coefficient in a bi-dimensional space. A surface plot and a heat map are provided in Fig. 4. Looking at the plots, a maximum in the RV coefficients can be noticed in the upper left quadrant, which is the one we chose to contain the maximum.