The first step in segmentation was preprocessing of the tomograms, which included noise removal and resizing.

Noise Removal: In order to remove noise from the images, we adopted the Noise2Void method proposed by [39]. Noise2Void uses convolution neural networks to learn about the properties of images and how to reduce noise without supervision. It aims to produce a regularized denoised image as output. We used the package in ImageJ to process the 3D X-ray tomograms. The Noise2Void can be downloaded at https://imagej.net/N2V.

Crop and Resize: Since the tomograms were collected by placing cells in the glass capillary, each slice was cropped by the capillary lines in the tomogram (S1B Fig). Cropping based on capillary walls creates different sized 3D images, so we resized the cropped images to a standard width, height, and depth of 280, 480, and 320 pixels, respectively.

In this work, we adopted an effective classical deep-learning model U-Net [12] for semantic segmentation. The main architectural design of the method is the combination of layer downsampling and upsampling, with skip connection between them as illustrated in S2 Fig. The first part is the contraction path (left side), also called the encoder, which is used to capture image context. The second part is the expanding path (right side), which is used to enable precise localization by deconvolution. Thus, the two sides combined form a U-shaped architecture that allows the network to propagate information collected from a larger receive field of images. To achieve more precise localization using multiple features, at every step U-Net uses skip connections that concatenate the output of the deconvolution layers with the feature maps from the encoder at the same level.

For our task, each U-Net uses an encoder that sequentially applies a convolutional block of two 3*3 convolution layers followed by a rectified linear unit (ReLU) and a 2*2 max-pooling with stride 2. At each decoding step, it adopts a deconvolutional layer that first interpolates the feature from its previous layer, concatenates with the corresponding skipped feature from the contracting path, and then goes through two 3*3 convolutions, each followed by a ReLU. Given the upsampled feature, we finally applied a 1*1 convolution layer on each feature element to get the predicted segmentation mask. To build the whole layer-by-layer process, we optimize the parameters of each layer, denoted as W, by minimizing the training objective: cross-entropy loss L_ce between the predicted mask y i ^ and ground truth manual segmentation y_i for each category i, using stochastic gradient descent. L c e = - ∑ i y i l o g ( y i ^ ) (1)

Since semantic segmentation was performed on 2D images, we merged the 2D labels into 3D organelle masks. To improve the performance, we predicted binary 2D labels from all three axes for cell, nucleus and mitochondria labels (S3 Fig). To generate the 3D masks of organelle, we developed a multi-view fusion algorithm that utilizes the complementary information to fix the uncertainty from the isolated prediction of three views. If the raw 3D volume data is represented as V ∈ R 3, our goal was to assign the label to each voxel v_i ∈ V. First, we construct 3D label masks by simply merging 2D labels in each view, i.e. V^X,V^Y, and V^Z. Each voxel v_i ∈ V in the raw tomogram has three predictions available from each view, corresponding to v i X ∈ V X, v i y ∈ V Y and v i Z ∈ V Z. Then, the final binary fused prediction is generated using the voting strategy as follows: v_i = 1 if v i X + v i Y + v i Z > = 2, else v_i = 0. Improvements in the segmentation accuracy is seen for the semantic masks of cell, nucleus, and mitochondria after the application of 3D fusion post-processing (S1 Table).

On a 4 TITAN Xp GPU setup it took 5–6 hours of training for the semantic segmentation on each U-Net (18 tomograms) and around 10 hours of predicting on each U-Net on a total of 132 tomograms.

Patch Split: Detecting the object granule (insulin vesicle) is challenging since the vesicles are small and tend to vanish in the max-pooling layer. Therefore, we decided to crop the raw 2D image (280 × 480 pixels) into several image patches before feeding them into the network. Each patch was about 150 × 50 pixels and overlapped about 30% with each other.

Mask R-CNN [15] is one of the state-of-the-art models for instance segmentation, developed on top of Faster R-CNN [14], which adopts the two-stage procedure shown in S2B Fig. The first stage generates the candidate bounding boxes for each object region. Specifically, the backbone network exploits a Deep Residual Network (ResNet) [40] with a Feature Pyramid Network (FPN) [41]. It consists of bottom-up and top-down pathways as well as lateral connections. The bottom-up pathway extracts features from raw images and the top-down pathway generates a feature pyramid map similar in size to the bottom-up pathway. Lateral connections merge feature maps of the same spatial size from the corresponding levels in the two pathways. This operation captures strong representative image features from multi-scale resolution images. Then, a lightweight neural network called Region Proposal Network (RPN) [14] identifies regions in the feature map that may contain objects, also called Regions of Interest (RoI).

In the second stage, the network predicts the classification output and refines the bounding box localization, as well as generating a segmentation mask at the pixel level. First, each RoI is pooled into a fixed-size feature map and then fed into two branches of predictions. The first branch predicts a class label and the bounding-box regression offsets for each RoI using fully connected layers (FCs). The second branch generates mask logits output per pixel for each granule at the pixel level.

Restoring the prediction to original image size: Since our prediction is based on image patches, these needed to be fused together to restore the original image size. In the overlapping area between two patches, the prediction with a higher score was retained.

3D Fusion: 3D fusion means we combine the instance segmentation from current 2D images to 3D instance. To achieve 3D fusion, we set an Intersection-over-Union (IoU) as a metric for the predicted segmentation mask on each 2D slice to determine if 2D instances belong to the same 3D instance. The instance IoU is calculated as the ratio of the overlapping area of the predicted masks to the total combined area from the adjacent slices. If the IoU is greater than the threshold of 0.5, the 2D masks are kept as the same instance mask, otherwise the masks are regarded as isolated instances.

On a 4 TITAN Xp GPU setup it took 15–16 hours of training for the instance segmentation with Mask R-CNN (18 tomograms) and around 20 hours of predicting on a total of 132 tomograms.

LAC is a linear function of thickness and chemical composition [42]. It reflects the absorption of X-ray samples as well as the molecular density with small regions. Regions with varying LAC were used to identify organelles in SXT data.

We set the priority for each label mask as: insulin vesicle mask >nucleus mask >mitochondria mask >cell mask. Masks with high priority replaced masks with low priority where they overlapped.

The Dice Coefficient is a commonly used metrics in semantic segmentation, which is calculated as follows: d i c e = 2 * | y i ^ ∩ y i | | y i ^ | + | y i | (2) Here | y i ^ ∩ y i | denotes the overlapping area between the predicted masks, y i ^, and the ground truth manual segmentation, y_i.

Recall, which measures how many of the actual positives our model captures, is defined as the number of true positives, T_p, over the number of T_p plus the number of false negatives, F_n. We define the distance function d as follows to measure the T_p. d = d u r y i ^ + r y i (3) where d_u is the Euclidean distance between the center of y i ^ and y_i. r y i ^ and r y i are are the radii of the predicted mask, y i ^, and manual segmentation, y_i, respectively. If d < 1, we treat it as T_p.

RDF was used to calculate the probability of organelle B’s appearance at a distance r from organelle A. [43] The probability g(r) is defined as: g A B ( r ) = ⟨ ρ B ( r ) ⟩ ⟨ ρ B ⟩ l o c a l = 1 ⟨ ρ B ⟩ l o c a l 1 N A ∑ i ∈ A N A ∑ j ∈ B N B δ ( r i j - r ) 4 π r 2 (4) where ρ_B(r) is the average density of B at a distance r around A. 〈ρ_B〉_local is the average density of B for all spheres around A. δ(r_ij − r) is the dirac delta function. In our calculation, insulin vesicle RDF is calculated on numbers since we have the instance segmentation on insulin vesicle masks. Mitochondria RDF is calculated on voxel distribution. Here we used RDF to divide cytosol into 8 shells, as shown in S2C Fig.

Insulin vesicle RDF are calculated by computing the center of each insulin vesicle instances distribution. Mitochondria RDF are calculated by computing the mitochondria voxels distribution. Insulin vesicle-mitochondria RDF are calculated by computing the center of each contact. A single contact is defined as a region that insulin vesicle voxels close to mitochondria surface with distance less than 1.5 voxels.

The first step in our method is the segmentation of single-cell tomograms (S1 Fig) to generate organelle masks for analysis using our auto-segmentation method, which combines semantic and instance segmentation frameworks(see Methods section for more detail). We used the U-Net [12] framework (S2A Fig) to train and segment the ‘cell’, ‘nucleus’, and ‘mitochondria’ labels and to obtain semantic segmentation masks. For ‘insulin vesicle’ labels we trained the Mask R-CNN framework [15] (S2B Fig) to obtain instance segmentation masks. In total, we used 132 tomograms in this study, out of which manual segmentation was available for only 24 tomograms. The 24 tomograms are randomly divided into three partitions: 18 for training, 3 for validation, and 3 for testing. The accuracy of the auto-segmentation was assessed using the test dataset and quantified using the Dice coefficient for U-Net [12] and the Recall for Mask R-CNN [15]. After segmentation evaluation, we applied the trained networks to the complete set of 132 single-cell tomograms and obtained 3D masks for cells and organelles including the nucleus, mitochondria, and insulin vesicles. The complete segmentation workflow is shown in Fig 1 with details provided in the Methods section.

To measure the accuracy of our segmentation results, we first computed the Dice coefficient and Recall for measuring the accuracy of the U-Net and Mask R-CNN networks for each of the three training datasets (Table 1), then we investigate the various features from our segmented masks based on linear absorption coefficient (LAC) and radial distribution function (RDF) (see Materials and methods section: Systematic analysis: Linear absorption coefficient(LAC), Radial distribution function (RDF)). The auto-segmentation results for Cell ID 766_8 are reported in Fig 2A and 2B together with the manually segmented masks.

The 2D mask predictions of cell and nucleus labels were quite accurate, with the mean Dice coefficient >91%, confirming high quality auto-segmentation. However, predictions made for mitochondria showed that the Dice coefficient only reached ∼70% accuracy. To investigate the relatively low accuracy of mitochondria masks, we first computed the contrast ratio, i.e., the normalized intensity value of the mitochondria mask in 2D slices divided by the surrounding cytoplasmic pixels in the manual segmentations (Table 2). The resulting low contrast ratio (approx. 1.1) revealed that mitochondria were only marginally distinct from the cytoplasmic surroundings, consequently explaining the lower accuracy of the mitochondria label prediction.

For the insulin vesicle instance segmentation results, the mean Recall was high with the value of ∼89%, indicating high prediction accuracy. However, the mean Dice coefficient and the mean AP₅₀ (average precision with an IoU threshold of 50%) only reached 32% and 41%, respectively. Note that the Dice coefficient was calculated using semantic masks converted from the corresponding instance masks, whereas AP₅₀ was computed using instance masks directly. Noteworthy, the contrast ratio of insulin vesicles in the tomogram of cell ID 766_8 was quite low (Table 3)), raising difficulties in both manual segmentation and auto-segmentation, indicated by low Dice coefficient and AP₅₀. Despite the low quality of this tomogram (will be discussed with more details below, Table 3, S4 Fig), we still achieved a Recall of 84%, confirming the high accuracy of auto-segmentation in reproducing manual segmentation results. For the tomograms of cell ID 784_5 and 842_17, manual segmentation results were nicely reproduced, indicated by the high Recall. Moreover, auto-segmentation predicted hundreds of additional insulin vesicle instances on regions that were not containing labels in manually segmented masks, indicated by low Dice coefficient and AP₅₀. This further justified the benefit of auto-segmentation in recognizing insulin vesicle instances that could not be recognized in manual segmentation (S5 Fig).

To investigate the relative low accuracy of mitochondria masks, we computed the contrast ratio and compared it with insulin vesicle, i.e. the normalized intensity value of the organelle mask in 2D slices divided by the surrounding cytoplasmic pixels in the manual segmentations. We observed that the contrast ratio of mitochondria was lower than that of the insulin vesicles (Tables 2 and 3), revealing that mitochondria are less distinct from the cytoplasmic surroundings, consequently explaining the lower accuracy of the mitochondria label prediction.

The instance segmentation of insulin vesicles using Mask R-CNN also provided the number of insulin vesicle for each cell (Table 1). We noticed in two of the three test datasets and on average that the insulin vesicle numbers from the auto-segmentation results were higher than the manually segmented vesicle numbers. This was mainly due to the fact that, in the manual segmentation process, clusters of insulin vesicles were counted as one single instance, whereas the trained network was able to discriminate single instances (Fig 2B1–2B3), further justifying the benefit of using instance segmentation for insulin vesicles. Moreover, we noticed in cell ID 766_8, the number of insulin vesicles is lower from the auto-segmentation result than from the manual segmentation result. This was due to the significantly low contrast ratio of insulin vesicles in the tomogram of this cell as compared to the other two cells (Table 3). For example, one organelle labelled as insulin vesicle in manual segmentation but not in auto-segmentation (S4 Fig) has a LAC lower than the normal range of insulin vesicles (0.3–0.6) [3].

Furthermore, we investigated the quality of insulin vesicle instance segmentation (see details in the Methods section). For each individual insulin vesicle, we used the instance mask and computed the average linear absorption coefficient (LAC) value [3, 44] of voxels at different radial positions inside the vesicle (Fig 2C1–2C3). We found that the LAC value increases from the vesicle boundary to the center, confirming the dense-core feature of insulin vesicles [3, 45, 46].

To further validate our pipeline, we investigated the subcellular variances under glucose and glucose + Ex-4 on the insulin secretion process and compared the results to those using manual segmented masks by White et al. [3]. Following the methodology presented in White et al. [3], we compared the variance on the same aspects of insulin vesicle LAC values, mitochondria LAC values and mitochondria volume (S7 Fig). We found that in both glucose and glucose + Ex-4 stimulations, especially in 5 min time point, there were significant increase in mitochondria LAC values and volumes as well as the insulin vesicle LAC values. Thus, results from our method confirmed the hypothesis that Ex-4 may affecting the trafficking system to promote insulin secretion, which was based on analysis on manual segmentation of SXT [3].

The process of insulin secretion took place in two stages. The first phase happened within the first 10 min of glucose stimulation, and the second phase lasted from 10 min to four hours [47, 48]. During the first phase, insulin vesicles docked at the plasma membrane [49] are immediately secreted, decreasing the number of pre-docked vesicles. Meanwhile, other insulin vesicles in the reserve pool are transported toward the plasma membrane in preparation to be secreted [35]. During the second phase, secretion of insulin continued at a low rate with newcomer insulin vesicles contributing to the secretion [50, 51]. These new vesicles were trafficked from the area within 0.6 μm of the plasma membrane [52].

We investigated insulin vesicle distributions during the secretion by comparing the probability of insulin vesicle occurring in cytosolic regions. We compared insulin vesicle behavior after stimulation with high glucose concentration (25 mM) at four different time points (5, 15, 30, and 150 min). The condition without glucose treatment served as a baseline control (Fig 3). In order to represent insulin vesicle distribution, we applied a RDF, which calculates the probability of insulin vesicles appearing at a distance r from the nucleus (Methods: Radial function distribution). We divided the cytosol (non-nuclear) space into 8 concentric regions based on the distance to the nucleus membrane (S2C Fig), with region r₁ closest to the nucleus and r₈ closest to the plasma membrane. Firstly, we found that at 5 min (the time point that falls in the first phase of insulin secretion), the probability of insulin vesicles appearing in regions r₇ and r₈ was lower as compared to the control (Fig 3). This showed that insulin vesicles in the readily releasable pool (RRP), which were docked close to the plasma membrane, were released quickly as the rapid reaction to glucose stimulation. Also, at 15 min (the earliest time point in the second phase of insulin secretion), we noticed a peak in region r₇ as well as a reduced probability in region r₈ compared to the first phase observation, showing that newly generated insulin vesicles were being transported towards the plasma membrane while docked vesicles were still being released. Later at 30 min, there was increased probability of insulin vesicles in the region r₈ due to the replenishment of RRP near the plasma membrane as secretion continued. Finally, at 150 min the probability of insulin vesicles in the region r₈ was highest due to the completion of insulin RRP replenishment.

Moreover, we found in the regions r₂ and r₃ that the increasing probability from normal (0 min) to 15 min represents the maturation of immature vesicles, while the probability in r₇ and r₈ were decreased referring to the docked insulin vesicles release. Our results were consistent with previous studies [50, 51], confirming the validity of the analysis using the auto-segmentation method. Later at 30 and 150 min, newly matured insulin vesicles were transported, either to be secreted or to replenish the RRP [53], causing a decrease in probability at r₂ and r₃. Thus, regions r₂—r₃ were associated with new insulin vesicle maturation (Fig 3, pink region), regions r₄—r₆ to transportation (Fig 3, cyan region), and r₇—r₈ to secretion (Fig 3, violet region).

We firstly compared distributions of insulin vesicles and mitochondria during treatments with glucose + Ex-4 (Fig 4A1–4A3). A lowered RDF in region r₈ (closest to plasma membrane) is seen within 5 min of treatment with glucose + Ex-4 as compared to the treatment with glucose alone (Fig 4A1). Thus, an increased number of insulin vesicles were released from the RRP with the Ex-4 treatment. In the previous section, we also observed that the RRP was replenished at 30 min, we found that co-stimulation with Ex-4 enhanced this effect. Additionally, the RDF of vesicles at 30 min in transportation regions r₄—r₆ was lower with co-stimulation with Ex-4 compared to glucose alone and was higher in secretion regions r₇—r₈ (Fig 4A1). These observations indicated that Ex-4 facilitated the rapid movement of vesicles towards the plasma membrane.

Moreover, as can be seen from Fig 4A2, mitochondrial RDF in regions r₄—r₆ was relatively higher at 30 min treatment with Ex-4 as compared to the baseline and the treatment with glucose alone. It has been proposed that the distribution of mitochondria was to match energy demand inside of cells [54, 55]. Thus, such aggregation of mitochondria provides a local energy source to facilitate the movement of vesicles under glucose + Ex-4 treatment. Additionally, we observed higher probability of contacts between insulin vesicles and mitochondria in the transportation region when co-stimulated with Ex-4 (Fig 4A3). Thus, we hypothesize that Ex-4 boosts the insulin secretion by recruiting mitochondria to the transportation regions. Such recruitment give higher priority to satisfying the energy demand of insulin vesicle transportation towards the plasma membrane.

As with Ex-4, we explored the effects of NN414 on insulin secretion by comparing insulin vesicle and mitochondria distributions during treatment with glucose + NN414 and NN414 alone (Fig 4B1–4B3). Since NN414 is a selective potassium channel opener and restricts insulin release, we also compared the NN414 condition with KCl treatment as KCl induces insulin secretion independent of glucose metabolism [56].

We found that at 30 min after treatment with NN414 alone, insulin vesicle probability was increased compared to baseline at regions r₂ and r₃ regions, which were related to insulin vesicle biosynthesis (Fig 4B1). After 30 min from treatment with glucose + NN414, we found that the RDF of insulin vesicle near the plasma membrane (regions r₇—r₈) was lower than the baseline and the treatment with glucose alone. Moreover, two significant differences in the RDF profiles under 30 min treatment with glucose + NN414 as compared to the 30 min treatments with only glucose and with glucose + Ex4: (I) the probability of mitochondria was lowered at maturation and transportation regions (r₂ to r₆) (Fig 4B2), (II) the probability of contacts between insulin vesicles and mitochondria were decreased at transportation region (Fig 4B3). These observations indicated that NN414 limits insulin secretion by detaining the recruitment of mitochondria and thus reducing the energy supply needed for the maturation and transportation of insulin vesicles (Fig 4B1).

In addition, we noticed that the mitochondria distribution changed drastically with NN414 treatment alone. Compared to cells with no treatment or with glucose + NN414 treatment, mitochondria in NN414-treated cells accumulated in the region close to the nucleus, as indicated by the increased probability in regions r₁—r₃ (Fig 4B2). Thus, we hypothesize that NN414 detain the recruitment of mitochondria regardless of glucose stimulation.