Two big challenges with membrane data are the presence of intensity inhomogeneities and punctuated gaps along the three-dimensional boundary. In Figure 1(A–C), we show a single cell membrane across the three cross-sections of , and . Intensity inhomogeneity (red and blue arrows) can be explained with help of an image formation model for membranes (Figure 1(D)). Here, optical planes (red lines) periodically section a dense cloud of fluorescent proteins tagged to membranes. The point spread function (PSF) of the optics accumulates emissions from a small neighborhood of fluorophores and creates intensity profiles shown in dark red. Mathematically, this is a convolution of the PSF with the fluorophore density function of the sample. The intensity at a voxel is therefore representative of the fluorophore density at the focused region of the tissue. Cell junctions are generally more intense as a result of high spatial concentration of fluorophores arising from the co-localization of multiple membranes. Additionally, membranes that are orthogonal to the imaging planes depict a crisp and bright intensity profile, whereas oblique membranes appear diffuse (observable in the and views in Figure 1(B–C)). This is because the PSF, as mentioned before, is shaped similar to an anisotropic Gaussian kernel with . The strength of the output signal is thus dependent on the relative alignment of he membrane and the PSF kernel. Thus, orthogonal membranes present a strong signal because the PSF samples fluorophores in the membrane “above” and “below” the focal plane. For oblique membranes, the space “above” and “below” is non-fluorescing so that output signal is weaker. In the limit, en-face membranes, especially those in between imaging planes are often very dim and difficult to detect even by the human eye.

In order to correct these two problems, we developed signal reconstruction techniques. Our algorithms are inspired by work on vessel-detection from MR and CT imagery in which Hessian-based filters were designed to detect vessels [18], [19]. They used the fact that eigenvectors of the image Hessian point in the directions of principle curvature. At vessel boundaries, the eigenvector with largest eigenvalue is almost normal to the boundary, and the one corresponding to the smallest eigenvalue points along the vessel axis. Here we extend these ideas to planar membrane structures and further improve these results using a voting strategy.

Our method has three stages (Figure 1(E)): (i) We observe that membranes assume locally linear intensity patterns especially in dense cell regions. This is used to design image processing filters to identify planar intensity formations in images. Automated scale-selection is accomplished by identifying the relative orientation of the membrane planes with a putative Gaussian PSF. (ii) Near inter-cellular junctions and due to image noise, the planarity assumption breaks down causing structural gaps to appear. The identified planar components are then used in a voting framework to fill gaps and eliminate spurious structures. (iii) After reconstructing membrane planes, we use the popular watershed methodology for image segmentation to identify cells.

Using our preliminary work in [20], we designed an image filter that responds to locally planar intensity structures such as those found in cell membranes and suppresses all other types of intensity patterns. The derived filter is based on the the Hessian matrix () of the intensity function combined with normalized Gaussian derivatives that provides an aspect of scale. Let , and be the sorted () eigenvalues of with corresponding eigenvectors .

In a local coordinate reference frame placed at a membrane voxel, we are interested in identifying the neighborhood intensity distribution. There are three types of distribution shapes that can be detected: rod, plane and ball (Figure 2). Based on our presentation, it is easy to see that a rod has a change in second derivative of intensity along its axis and maximal change in the cross-sectional plane. Hence, and points along the axis. and lie in the cross-sectional plane with . In the case of a plane, the maximal change in directional second derivative is along the normal and there is no change in the plane. Hence, we have along the normal and and lying on the plane. Correspondingly, the eigenvalues follow the relation . Finally, for a ball (uniform signal), there is no preferential orientation and all directions have the same change of directional second derivatives. Hence, . Table 1 summarizes these different cases.

In order to detect membrane structures, we want to selectively identify pixels that belong to a plane distribution rather than a ball or a rod. Hence, we define the planarity of a voxel as the similarity of the local neighborhood to a plane-like structure, as:(1)(2)

Here, with larger values indicating more similarity to a plane at regularization scale . The free parameters are set to 1 in our experiments and code but may be fine-tuned depending on the specifics of the imaging modality. For the case of bright membranes on a dark background, a positive denotes background and hence the planarity output is set directly to 0. We now explain how background voxels and voxels corresponding to the rod and ball forms are suppressed by design:

A heat map of the sampled function in the space with all free parameters set to is shown (Figure 3). Since is a three dimensional function defined on (, , ), we show a single cross-section at . In the figure, high filter response corresponds to voxels having a plane form alone (), while background voxels and voxels corresponding to the rod and ball forms are suppressed.

Earlier, we described how membranes have crisp or diffuse profiles depending on their orientation with respect to the optical planes (Figure 1). In an ideal situation, the PSF is an impulse-response and the membrane plane is infinitesimally thin and is sufficient for its detection. However, this ideal is not achievable in optical microscopes. So, we model the PSF as an anisotropic Gaussian kernel (). The signal for an orthogonal membrane is contained in a small band of pixels as opposed to an en-face membrane which is diffused farther out. Thus, the scale of Hessian computation () needs to adapt depending on membrane orientation. Let and represent the optimal scales for orthogonal and en-face membranes with respect to the optical plane with normal (a unit vector along the optical axis). The normal orientation of membranes locally is given by the eigenvector , as per our convention. Therefore, we automatically determine scale as:(3)

The dot product of and returns the cosine of the angle between the optical axis and the largest principal components, so if they are orthogonal and if they are parallel. In our case, for a orthogonal membrane we set and , which makes . To first determine membrane orientations ( in the above formula), we use a blind scale determined by to compute .

In Figure 4, we show a three-dimensional result of applying the planarity function on raw data (a–d). The result (e–g) is displayed along orthogonal sections in , and respectively. The last column is a detailed view of the first column of images. We identify membrane voxels inspite of severe intensity inhomogeneities and noise. The image center shows cells in the notochord region which have a very weak intensity profile but were uniformly identified by our method. Local variations of membrane intensities due to orientation differences with the optical planes are also compensated. This can be seen in the orthogonal planes and , where membrane structures are well-reconstructed. Upon zooming in at a high resolution in (h), we spot several gaps in the membrane structure especially near membrane junctions. At these locations, the intensity structure ceases to be of planar distribution. In some locations, planar noise patterns create false positives in detection. In order to eliminate these spurious structures and reconstruct membranes alone, we use the tensor voting framework to build on the output of the planarity filter.

The principle of tensor voting is that image voxels vote in their surrounding neighborhood to propagate information about the presence of a surface passing through them [21], [22]. At each voxel, votes are cast and accumulated in a local neighborhood. The basic idea behind this process is that if a set of unconnected voxels exists on a geometric surface oblivious of each other, then by voting each voxel develops a sense of direction and affiliation. Thereafter, the surface boundary can be automatically extracted by a region segmentation procedure such as the watershed. The geometric surface in our context refers to the membrane planes.

The application of tensor voting to membrane images has previously been considered. Loss and colleagues developed an iterative extension of the tensor voting framework to demonstrate its application on low fidelity membrane images [23]. Although, tensor voting methods are parameter-free, they are computationally expensive and do not scale well with large image sizes. The iterative extension exacerbates the computational cost. In our case, the accurate detection via the planarity filter provided to the tensor framework eliminates the need for iterative methods. In another extension, Parvin et al. [24] developed an iterative voting system that employs tunable kernels to refine paths of low curvature in images. Given the short nature of membrane segments, we do not consider the iterative extension here.

There are three stages of the tensor voting process: (i) initialize a tensor image, (ii) cast and accumulate votes at each voxel, and (iii) extract membrane saliency image.

We use the watershed algorithm for obtaining high quality segmentations once the reconstruction procedure is completed [11]. We use the saliency images generated from tensor voting as topographic maps in the watershed procedure. The saliency image results from the votes of tensors oriented along the membranes thereby causing a very rapid change in values normal to the membrane. Figure 5 shows the resulting cell segmentations obtained from using our new approach on two timepoints of noisy membrane data. The output of the high-quality reconstructed membrane signal is shown in the orthogonal image planes. A step-wise graphical overview of the complete segmentation process is provided in Supplementary Figure S2.

Since there is no gold standard available for evaluating algorithm performance, we synthesized membrane images based on an image formation model that simulates confocal microscopy of membrane labeled embryos (Supplementary Section S3 and Supplementary Figure S3). The advantage of using synthetic images is that ground-truth is exactly known and different imaging parameters can be rapidly tested. A spectrum of ten images with varying noise parameters was generated for comparison, and the performance of the algorithm is described in Table 2. Despite the fact that cells are tightly-packed and large additive noise is present, the proposed method reconstructs membranes and segments the touching cells with high precision. Figure 6(A–C) shows an example of a synthesized membrane image containing cells with the corresponding segmentation results shown in Figure 6(D–F). The performance of the algorithm steadily degraded for higher levels of noise as expected. It was observed that in the worst case, we obtained a precision of 94% and recall of 98%. The enchroachment on neighboring cells was limited to 1.42 and with an overlap of more than 84%. As it is clear from these results, our proposed segmentation method achieves significant volume overlap with the ground truth, indicating the accurate performance of the segmentation method.

We next applied the method to images of zebrafish mesoderm obtained at 12 hpf (Figure 4). Four membrane images were used to evaluate the proposed segmentation method. Using the publicly available GoFigure2 software, an expert manually marked all the somite cells in a small image region by drawing 2D contours on different image planes. For each cell, a 3D mesh was generated out of the sampled 2D contours by using an automatic surface reconstruction procedure. The 3D meshes were used to compare and assess the performance of the automated segmentation algorithm. To demonstrate the effectiveness of our reconstruction procedure for automated segmentation, we compare the performance of three versions of the automated algorithm, namely:

In Table 3, we evaluated the segmentation metrics and observed that the basic algorithms 1 and 2 suffer from a high over-segmentation (13.06%, 7.51%) and under-segmentation (13.48%, 5.92%) error rates. Over-segmentation occurs when spurious structures are present in cell interiors that split single cells into multiple labels. Under-segmentation rates are high due the lack of membrane pixel connectivity especially in and planes. When segmentations correctly matched, the algorithms localized the boundary accurately (0.44 and 0.42 ) and also had a significant volume overlap (89% and 91%). In contrast, algorithm 3 shows significantly improved performance. On average, the over-segmentation and under-segmentation rates are 4.66% and 3.3% respectively. For the matched set of cells, the average volume overlap and L2 Hausdorff distance are 93% and 0.27 , respectively demonstrating the low distortion in object morphology. Our results indicate that the reconstruction procedure enhanced membrane connectivity and eliminates spurious structures, thereby reducing the over and under-segmentation error-rates.

The two scale parameters and constitute two important parameters in generating the automated output. Thus, we explored a range of and values and assessed the variation in the performance of our method. While changing a given parameter, we ensured the other parameters were optimally set. Figure 7 reports the average precision and recall values plotted against / values for the four manually segmented datasets. We observe robust performance for a broad range of and parameters ([0.7, 1.5]) with a gradual degradation in performance. High values () tend to assign more importance to membrane smoothness over large scales. This negatively impacts membrane connectivity at cell junctions leading to under-segmentation and hence lowers the precision/recall metrics. Low values () tend to localize membranes more accurately but retain spurious structures that causes over-segmentation and also lowers the metrics. Hence, a judicious choice that balances over-segmentation and under-segmentation rates is recommended.

We also applied the method to three zebrafish images in which nuclei and membranes are imaged in separate channels and segmented separately. For nuclear segmentation, we use an improved version of the watershed algorithm using seeds [26]. Although the nuclear and membrane segmentations are not perfect, in an ideal scenario there should exist a one-to-one correspondence between both segmentations (Figure 8). We extract the centroids of cells and nuclei and match them using a nearest neighbor method. Table 4 provides the details of the matching. On average, the number of nuclei extracted are less than the number of cells from membrane information. This discrepancy is because interstitial space in the tissue (or vacuoles) can be segmented as cells even when they do not exist. These empty spaces can be difficult even for a human to distinguish in the absence of any other information. Our experiments demonstrated an excellent match between the nuclear and membrane segmentation algorithm outcomes indicating a robust performance of our segmentation software.

During zebrafish somitogenesis, a series of epithelial tissue blocks forms rhythmically by separating from the presomitic mesoderm tissue (PSM) [27]. A total of 28 pairs of blocks known as somites sequentially form beginning at 10 hpf with a period of approximately 30 minutes [28]. Somites are formed by cell sorting from the PSM. Each somite is structurally composed of epithelial (E) cells on the boundary with an inner mesenchymal (M) core. Throughout somite formation, the PSM maintains a steady-state by coordinating the anterior process of somite formation with cell recruitment and proliferation at its posterior end. The PSM is gradually patterned along the anteroposterior axis by cellular rearrangements and tissue/cell-shape changes, deriving its input from an oscillating molecular circuit known as the segmentation clock (not to be confused with image segmentation) [29], [30]. Segmentation clocks operating inside individual cells are synchronised along the PSM to create periodic waves of oscillating gene expression. While there has been substantial progress in understanding the molecular mechanisms of wave initiation, synchronization, and the readout circuitry, the cellular and mechanical mechanisms involved in physically sculpting a somite are not clear due to the lack of high-quality image data and subsequent robust analysis [31]. For example, it is not exactly known how the sorting interface develops, what the cell movement patterns at the interface are, how many cells are involved, and what the corresponding changes in cell and tissue morphology are.

Therefore, our goal was to obtain time-lapse membrane images during somite formation, apply our reconstruction techniques, and quantify cell dynamics. We chose to in toto image the formation of somites 3, 4, and 5 in a zebrafish embryo mounted dorsally, with a 40× objective, and with a time-sampling of 2 minutes over a period of 60 minutes using confocal microscopy [32]. The beginning marked the formation of somite 3 with a discernable interface with the presomitic mesoderm. During the time-lapse, we observe the complete separation of somite 3 and 4. Somite 5 forms a discernable interface at the end of the time-lapse thus completing two full cycles of segmentation. In Figure 5, we present the results of our reconstruction (orthogonal sections) and automated segmentation (3D view) of the PSM cells at 3 and 5 somite stages (ss). Automated segmentations were overlaid on the reconstructions to show the excellent agreement in contours. We then proceeded to analyze the formation of somites by quantifying differences between 3 and 5 ss (Figure 9). As a consequence of somites physically separating and becoming spherical, interface surface area decreases across all the three somites. Given that somite 3 is farther along than somite 4 and somite 5 in the process of forming round somites, their surface areas (see blue and red bars) are monotonically higher. In particular, somites 4 and 5 show a large surface area in the PSM initially (see blue peaks). At ss 5, somite 3 and 4 show significantly smaller surface area due to the completion of somite rounding while somite 5 is still halfway through. Thus, the blue peak at somite 3 roughly corresponds to the red peak of somite 5 given the same relative progress into somite formation. The total number of cells is very consistent across the three somites (Figure 9D).

In order to understand the corresponding changes in cell parameters, we then identified the number of epithelial and mesenchymal cells in the formed somites. Mesenchymal cells which do not touch the surface of the somite were a small fraction (5–20%) of the cells (Figure 9D). The epithelial layer formed a single layer of cells over the mesenchymal cells. We labeled these cells in different colors and retrospectively tracked their location into the PSM. For tracking, segmentations at individual time-points were linked based on optical flow-fields that were first reconstructed [33]. Errors in the tracking were corrected manually using the publicly-available GoFigure2 software (www.gofigure2.org). We then computed the principal diameters of the cells along their principal axis. A typical workflow consists of first visualizing and interactively exploring the distributions of cell shapes, sizes, and locations after the segmentation process is completed. The process of interaction involves zooming in and out, changing the viewing angle, hiding a subset of data, and visualizing specific outliers or data points. Scatterplots in Figure 9(E–F) show the distributions of the epithelial (blue) and mesenchymal cells (red). In ss 3, we observe large and homogenous variation of cell morphology across E and M classes. At ss 5, M cells form a more narrow distribution while E cells spread out to form diverse cell shapes neccessary for constructing spherical somites with continuous epithelia. A few examples of such cells are shown in (F). After finding interesting correlations and trends, accurate 2D scatterplots or figures can be used for effective visualization. Hence, we also computed changes in cell volumes (size) in Figure 9(G–H). Here, we observe that the distribution of mesenchymal cell volumes (red) narrows and interestingly we find that the mesenchymal cells shrink in volume as the somite forms. We are now analyzing somite formation rigorously across all intermediate time-points, combined with tracking results for individual cells and across multiple datasets. As part of our future work, we plan to integrate the process with the underlying molecular circuitry to obtain a multiscale view of somite formation.

Our work successfully demonstrates the utility of our algorithms in enabling the quantification of cell shape and size, tissue interface areas and volumes, and reconstruction of cell lineages and fate maps by tracking segmented cells. By recovering individual cell dynamics and their collective behavior in tissue from time-lapse images, a deeper understanding of the mechanisms involved in morphogenesis can be obtained. Thus, our algorithms are computationally robust and can be deployed to facilitate the analysis of a wide-variety of morphogenesis systems.