Multiscale filtering is an appropriate choice, when nuclei have size variations with respect to time and space. Bashar et al. proposed such an approach, where the minimum and maximum filter lengths () required for spatial adaptation is computed from the current and all of the previous time-point images [19]–[21]. Normalized volume ratio of the candidate object pixels was used to obtain above lengths. However, the inclusion of background pixels in the candidate regions especially in the case of touching nuclei limits the improvement of the detection results by 2 on average. In this study, we estimate the above lengths based on the extracted nuclei objects through optimization as detailed in the section Nuclear Size Optimization. In a single-embryo image, if be the total nuclei and is the optimal diameter (See Eq. 15) for the th nucleus, we can obtain the minimum and maximum object sizes by(3)(4)

Once the maximum and minimum object sizes were known, we can compute and by(5)(6)where is the weighting factor, varying from 0 to 1. It is chosen 1.0 for the selected Gaussian filter. Note that , are the maximum and the minimum filter lengths, used for processing the first frame in the sequence. At any time-point t, if and are known, we can obtain multiple filter lengths for the spatial optimization using(7)where ; , and is the fixed incremental length. With the above lengths, multiple filters are applied at every pixel of the 3D image using Eq. 1 and the resulting responses are maximized by(8)

Spatio-temporal adaptation described above provides stable nuclei detection, which is insensitive to the initial filter parameters.

If nuclei in a single time-point 3D image have very small variations in sizes, single-scale temporal adaptation is an alternate option. In the single case, the enhanced image in Eq. 1 can be given by(9)

Once again, we select Gaussian function as our base filter, whose spread parameter () in the separable and isotropic case, can be related to the finite length parameter () by . This length should corresponds to the mean size of the nuclei for the adaptive enhancement. In our study, an initial value () of this parameter has been chosen empirically. Once we have at hand, we can run the proposed size optimization step (Please refer to the section Temporal Feedback Mechanism) to estimate the average object diameter () from all nuclei of the image at time . If a single-embryo image at time have nuclei and is the optimal diameter (See Eq. 15) for the i nucleus, we can compute(10)where and is the weight factor as described before.

Note that all of the animal experiments were performed with the approval of the Animal Care and Use Committee of Osaka University, Osaka, Japan.

Figure 6 shows the results of the estimated nuclei for four developing mouse embryos. Image sequences in these embryos cover mouse development until early blastocyst stage. Each embryo shows a consistent development from 1-cell stage to approximately 32 cell stage. Note that the estimated nuclei at or before the first discontinuity (at around frame s32) in each graph represents pronuclei of the two sex cells (i.e., male spermatozoa and female oocyte) at the zygotic 1-cell stage. Therefore, one cell stage of fertilized cell is not distinguishable from confocal images. The beginning of two-cell stage starts at around s41 and then undergo systematic cell divisions through multiple developmental stages (e.g., 2–4, 4–8, 8–16, or 16–32 cells). Above figure shows almost constant populations at different developmental stages, providing an indication of healthy embryos. It also indicates the efficiency of the proposed detection method. Note that the transition periods gradually increase towards larger populations, implying the asymmetric cell divisions of the early developing embryos. Another observation is the more frequent ups and downs in the nuclear count graphs at higher developing stages (e.g., 32-cells). This behavior indicates the increasing difficulties in detecting clustered cells at larger populations in an automated way.

Figure 7 shows volume rendered snapshots of the extracted nuclear centroids. The top row of Fig. 7 shows nuclei volumes, obtained by manual threshold, while the spherical color objects in the bottom row indicate the centroid detection ability of the proposed method, even when the nuclei are highly clustered. Figure 8 shows volume rendered views of the segmented nuclear volumes by manual (column A), typical (column B, [22]) and the proposed (column D) methods, respectively. Note that the proposed centroid-driven segmentation method successfully separates the clustered nuclei, even though there are lots of overlaps among closely located nuclei at high time-points (e.g., t76, t86, and t96).

Results of nuclei detection have been applied for tracking nuclei using a simple frame by frame association method based on nearest neighbor searching technique [29]. Figure 9 shows the preliminary results at 8-cell (Fig.9 A–C) and 16-cell (Fig. 9 D–F) stages, respectively. Twenty two (i.e., t1–t22) and twenty six (i.e., t36–t61) frames in the 8-cell and 16-cell stages were used in this study. Cell movement patterns were shown in the 2D projection space (Fig. 9 (A, D)) as well as in the 3D space (Fig. 9 (B–C, E–F)). A good agreement between the estimated (blue lines) and the manually generated tracks (red lines) indicates the effectiveness of our tracking method. The color gradients of the tracklets in Fig. 9 (C, F) express the moving directions of nuclei within the embryo, where blue and red lines indicate the start and end points of their journey. However, small translational error between the estimated and the GT tracks do exists, which implies the subjective error of the ground truth data.

Quantitative analysis is conducted using nuclei counts and F-measure metrics. We compute these metrics from 101 3D sequential images of a developing embryo that contains 8 to 32 cells. Figure 10 shows the results of the estimated nuclei over time. Blue lines in this figure show the estimated nuclei by the proposed method and its variations, while red lines show the ground truth nuclei. Correlations between blue and red curves roughly indicate the degree of correct detection by the proposed method and its variations. The proposed adaptive method showed clear superiority as the estimated lines closely follow the GT lines without much perturbations. The single scale version of the proposed method, i.e., Variant-1 also follows the GT curve except at the large population stage (e.g., 32 cells), where frequent ups and downs were observed. Variant-2 and Variant-3, which are the simplified versions of the proposed method, show more frequent oscillations in the estimated nuclei numbers that we observe throughout the nuclei-count curves. Note that there is spike in the ground truth at frame number 9 as shown in Fig. 10. This spike indicates ten nuclei at frame-9, while neighboring frames have constantly eight nuclei. In fact, the correct number will be eight, since the timing does not confirm any real cell division event. A close inspection however showed that it was a mistake happened during visual inspection.

Similar findings are reflected in the F-measure graphs (Figure 11), computed on the estimated nuclei. Very high values and less fluctuations of this metric, especially at high time-points, indicate the efficacy and stability of the proposed method and its single-scale variant. F-measure time-series for other variants (Variant-2 and Variant-3) show gradually increasing perturbations from multiscale to single-scale versions. These results are in congruence with the nuclear counting graphs in Fig. 10. One reason behind these observations is that the simplified methods (Variant-2 and Variant-3) use object-size threshold (), which sometimes removes small low-contrast objects with diameters less than 10 pixels (i.e., pixels). On the other hand, threshold operation in the proposed method (and its variant) might have a very small effects on the estimated filter lengths. However, such small perturbations do not affect the enhancement and subsequent detection steps in these methods. Both experiments also indicate that the multiscale enhancement based methods (proposed and Variant-2 Figs. 10 and 11 (A) and (C)) produce better results (in terms of nuclear counting and F-measure) than their single scale counterparts (Variant-1 and Variant-3 Figs. 10 and 11 (B) and (D)).

The efficiency of the proposed method and its variants can be computed from the grand average values of the performance metrics as shown in Fig. 12. Note that the three estimated metrics (sensitivity, precision, and F-measure) are averaged over temporal (101 time-points) and parametric (Please refer to the next section) axes. On average, F-measure scores (green bars) of the methods in descending order are (i) Proposed (99.31), (ii) Variant-1 (99.13 ), (iii) Variant-2 (98.74), and (iv) Variant-3 (98.53 ). These results showed very competitive performances of the proposed method and its variants. Therefore, the use of the simplified methods may be an alternative, when users require handling a reduced set of threshold parameters. Note that multiscale enhancement-based methods (Proposed and Variant-2) showed slightly improved performances over their single scale counterparts (Variant-1 and Variant-3). This improvement was expected because nuclei sizes usually have intra- and inter-frame variations for which multiscale filtering usually works well.

Estimating nuclear volumes is very important for various applications including tracking [15] and cellular phenotype analysis [4]. An experiment with 91 sequential 3D images (t1 to t91) is performed to estimate nuclear volumes applying the proposed method in Fig. 4. Estimated nuclear diameters and their variations are shown in Fig. 13. Two important observations are: (i) our datasets have small but smooth variations of average nuclei diameters over time (Fig. 13(A)); (ii) nuclei diameters have large variations during the occurrences of cell division events (i.e., t21–t35, t62–t79) compared to other times (Fig. 13 (B)). Therefore, standard deviation of the estimated nuclei sizes can be used to quantify gross transition periods of the cellular development in the multicellular embryos.

Expected outcome of the proposed method should be insensitive to the selection of the major parameters. Robustness to parameter variations is therefore an important consideration. Once again, sensitivity, precision, and F-measure metrics are used to evaluate parameter sensitivity. In our enhancement-based method, we consider single/multiscale filtering parameters (i.e., the average or the minimum and maximum filter lengths) are main parameters that can affect final detection results. To justify the robustness of the proposed method (and its variants), a set of parameter values is selected on the basis of the largest nucleus in our dataset. We obtain this size ( 25 pixels) from a 3D image having fewer objects using simple threshold method. For multiscale based methods (Proposed and Variant-2), the minimum and the maximum filter lengths ( and ) were varied from 5 to 50 and 17 to 62 pixels, respectively. A fixed interval (empirically chosen) of 12 pixels is maintained between each pair (minimum, maximum) of parameters (e.g., (5, 17)). A set of parameter pairs is thus computed applying linear sampling on both parameter ranges using 5 pixels interval. Filter lengths for multiscale filtering (spatial) within each pair is obtained using a fixed sampling interval, , of 3 pixels (Eq. 7). For single-scale based methods (Variant-1 and Variant-3), initial filter length () is varied from 5 to 50 pixels with an increment of 5 pixels.

Analysis results were shown in Fig. 14, where blue, red, green, and purple curves show variations of sensitivity, precision, and F-measure metrics for the proposed method and its variants. Clearly, all three metrics for our adaptive methods preserve high accuracy ( 98 ) despite having a large variations of filter parameters. However, sensitivity (Fig. 14(A)) and F-measure (Fig. 14(C)) curves of the previous methods start dropping at (25,37) (multiscale) and 25 (single-scale) and ultimately reach to the 83.71 and 90.20 , when filter parameters reach to the high end of the selected range. Above observation is re-confirmed by the F-measure plots at the highest parameter settings (Fig. 14(D)), when previous methods drop to 90.20 and 90.35, while the proposed method and its three variations maintain their high detection rates, i.e., 99.31 , 99.13 , 98.77 , and 98.56 , respectively. Precisions for all methods remain in the high states as expected because over-smoothing by large filter lengths reduces both the true and the estimated nuclei without increasing false detections (Fig. 14(B)). Performance degradation due to the wrong selection of filter parameters can be well explained by Fig. 15. All metrics (i.e., nuclei-count, sensitivity, and F-measure) except precision (red and green lines) start dropping at t31 and never return to the high state for previous methods (Fig. 15 A, C, D). In contrast, the proposed method and its variants (blue lines) always maintain high metric values except slight drifts at higher time-points. Precision remains in the high state for all methods except some minor variations.

In order to justify the noise tolerance of the proposed method, additive white Gaussian noise is added to a 3D image before applying nuclear detection algorithm. The considered image is a snapshot of a developing embryo at the early blastocyst stage containing 32 nuclei. If is the image and is the noise signal, generated from the noise spread , then signal-to-noise ratio (SNR) in dB and the noisy image () are obtained [30] by (19)(20)

To cover a large SNR range, i.e., 104.74–4.74 dB, we compute 30 noise spread values () linearly using , where for the whole set of values, while for the first 21 and 100.0 for the last 9 values. Our proposed method is then applied to the noisy image () to recover nuclei objects. A sample result is shown in Fig. 16, where each column represents a noise (SNR) level, noisy image, its histogram, corresponding enhanced image, and its histogram, respectively.

First row shows that the image is gradually degraded with increasing noise variances, i.e., the decreasing SNR levels. Nuclear structures get quite damaged at the highest noise level, i.e., at dB. Histograms in the second row indicate that the widths of the intensity distributions are expanding slowly with decreasing bin frequencies at increasing noise levels. This is what we expected since Gaussian noise randomly adds discrete black and white dots, which generates new intensities by breaking original intensity structures. However, our robust method recovers all objects by the adaptive multiscale filtering (Please refer to the third row of Fig. 16). Fourth row shows the histograms of the enhanced images that are almost same in structures, indicating the robustness of our method irrespective of the noise levels in the input images. Figure 17 shows the full experimental results, which include the graphs for SNR levels, ground truth nuclei numbers (light blue), and the estimated nuclei against the noise spread values. This figure shows that the selection of wrong parameters to large values (e.g., and ) degrades the detection performances of the previous methods when cell population increases to 32 (green and purple lines). Note that the detection accuracy falls below 75, i.e., less than 24 nuclei are detected out of 32 nuclei, which further decreases due to increasing noise spreads (Fig. 17). In contrast, the proposed adaptive method quickly re-estimate the required parameters to (11,31) at 32 cell stage even though the initial parameters for the first frame were large, i.e., (50, 62); It achieves high detection accuracy 100 , which remains almost steady irrespective of the noise levels in the input image (red line). This is the beauty of the proposed method because its multiscale spatio-temporal adaptive filtering recovers almost all nuclei through optimal enhancement.

Comparative analysis is summarized in the Table 1. Proposed method and Variant-1 show steady high F-measure (99.31 , 99.13 ) for all sets of initial filter lengths (Columns 2 and 3). F-measure metric values for Variant-2 and Variant-3 are smaller at the lowest end (98.47 , 98.32 ) than those at the highest end (98.77 , 98.56 ) of the parameter range (i.e., (5,17) (50, 62) for MSF and (5 50) for SSF) indicating their sensitivity to small filter lengths. On the other hand, F-measure scores of Previous-1 and Previous-2 fall drastically with increasing filter lengths from their high values (99.30 , 99.20 ) to the lowest values (90.20 , 90.35 ), indicated as boldface numbers in the table. At this condition, our proposed method can achieve 9 improvement over the previous methods. Note that the previous methods can also achieve good accuracies (99.30, 99.20) if the filter lengths are properly set to 15 for single-scale and (20,32) for multiscale cases (boldface numbers in the table). The robustness of the proposed method (and its variants) is due to the fact that the feedback mechanism computes proper filter lengths, which ensure accurate enhancement of the preprocessed image. As a result, unwanted local peaks even within nuclear region get suppressed and homogeneous objects are produced. In the non-adaptive case, there is no mechanism that can revert unexpected structures induced due to the improper enhancement.