Let I t be the gray scale image at time t and assume that N animals in total need to be tracked. Prior to further image analysis we compute a static background image which includes almost all immovable artifacts. Since images produced by FIM have a black background with bright foreground pixels and since we assume that an animal moves more than its own body length during the recording, the calculation of the background image B can be done using the minimal pixel intensity value over time, resulting in B r , c = min t I t ( r , c ) where I t ( r , c ) is the pixel intensity at row r and column c at time t.

Subsequently, the foreground image F t containing almost all objects of interest without the artifacts present in the background image B is obtained by F t ( r , c ) = I t ( r , c ) if | I t ( r , c ) - B ( r , c ) | ≥ τ B ( r , c ) 0 otherwise with τ B ( r , c ) = τ + B ( r , c ) where τ is a user set gray value threshold. Given F t the contours c t i of the animals are calculated by using the algorithm proposed in [14] resulting in a set of contours C t = { c t 1 , c t 2 , . . . , c t N t }. N_t might differ from N since animals can be in contact with each other (leading to merged contours) or impurities on the substrate which are not included in the background image lead to artifacts. However, the contours in C_t can be filtered to identify single animals by assuming that all imaged animals cover approximately the same area. The filtered set C ˜ t of contours is given by C ˜ t = c t i | λ min ≤ A ( c t i ) ≤ λ max , i = 1 , … , N t where λ_min < λ_max are two user defined thresholds and A ( c t i ) is the contour area given by the number of pixels enclosed by c t i. Both the contours that fulfill A ( c t i ) > λ max which are assumed to represent colliding animals and contours with A ( c t i ) < λ min which are assumed to be artifacts are ignored in further calculations.

For each contour c t i ∈ C ˜ t we compute a model representation of the associated animal. First, the spine is calculated based on curvature values which are obtained for each point p on the contour using the first pass of the IPAN algorithm [15]. Given all curvatures ∠_p the two regions with sharpest acute angle are located using a sliding window approach as illustrated in Fig 2a. The point with the sharpest overall mean angle is identified as head h and the point with the second sharpest mean angle (with an appropriate distance δ_{h ↔ t} to h) is identified as tail t. This assignment is done during the model extraction for each frame individually. In order to avoid head and tail switches and to ensure a reliable identification of these points for different organisms like larvae or C. elegans, the positions of the head and tail are refined in a post processing step using posture and motion features over time (see Post processing section).

The initial h and t identification is used to split the contour into two halves c t , 1 i and c t , 2 i. Without loss of generality, let | c t , 1 i | < | c t , 2 i |. Then c t , 2 i is re-sampled, so that | c t , 1 i | = | c t , 2 i | by utilizing linear interpolation between points in c t , 2 i. Now each point p 1 j ∈ c t , 1 i corresponds to a unique point p 2 j ∈ c t , 2 i. The spine points s₁, …,s_{L − 2} are calculated by determining L − 2 equidistant pairs ( p 1 j , p 2 j ) along c t , 1 i and c t , 2 i and by setting s j = p 2 j + p 1 j 2. The radii r_j are calculated similarly by r j = ∥ p 2 j - p 1 j ∥ 2. In addition, the center of mass m is calculated based on c t i. As a result, each animal i at time t is defined by a model l t i = ( h , ( s 1 , r 1 ) , … , ( s L - 2 , r L - 2 ) , t , m ) (1) which is depicted in Fig 2b.

Considering a sufficient spatio-temporal resolution tracking can be done by assigning animals between consecutive frames: all active animals L t = { l t i | i = 1 , … , N t } at time t need to be associated with either one detected animal from the set L t + 1 = { l t + 1 j | j = 1 , … , N t + 1 } at time t + 1 or removed from the set of active animals. Mathematically this assignment is known as a bipartite matching between L t and L t + 1. Let the costs for an assignment between an animal i at time t and an animal j at time t + 1 be κ_ij. This leads to the following cost matrix: C t = l t 1 l t 2 ⋮ l t N t ( l t + 1 1 l t + 1 2 ⋯ l t + 1 N t + 1 κ 11 κ 12 ⋯ κ 1 N t + 1 κ 21 κ 22   ⋯ κ 2 N t + 1 ⋮ ⋮ ⋱ ⋮ κ N t 1 κ N t 2 ⋯ κ N t N t + 1 ) ∈ ℝ N t × N t + 1

The following three cost measurements are implemented in FIMTrack:

Since the center of mass is extracted from the contour, the translation between consecutive frames is very smooth. In contrast, the middle spine point contains more jitter, but is forced to be inside the contour. Costs derived from the overlap of two consecutive contours fulfill both criteria as they are smooth and based on pixels within the contours.

To solve the aforementioned assignment problem two algorithms are implemented in FIMTrack. One is the Hungarian algorithm [16, 17] which has one drawback: Given distance-based costs, the algorithm will find assignments for all n ˜ = min ( N t , N t + 1 ) animals. For example, if animal i disappears at time t while another animal j appears at time t + 1, the algorithm will assign these two animals even if the Euclidean distance between them is very large. Thus, we check each assignment if at least one point of the model of animal i is inside the contour of animal j. Otherwise animal i is considered to be inactive, the associated trajectory is terminated, and animal j is initialized as a new animal.

The second algorithm follows the greedy pattern. Given L t and L t + 1 the algorithm determines sequentially for each animal l t i ∈ L t the best matching animal l t + 1 j ∈ L t + 1 given one cost measurement (note that each match has to be unique). To exclude irrational assignments this algorithm requires an additional threshold τ_greedy specifying the maximal distance between two consecutive points if distance-based costs are used or the minimal amount of overlap in case of contour-based costs.

The possibility to choose between multiple costs and optimization algorithms extends the range of organisms which can be analyzed even at various spatial and temporal resolutions. For example, during peristaltic forward locomotion of Drosophila larvae, the Hungarian algorithm in combination with overlap-based costs is feasible to associate the larvae over time (Fig 4a). In contrast, rolling larvae with unsuitable temporal resolution lead to false assignments using the Hungarian algorithm with overlapping contour costs: due to strong changes in the body bending and the relatively fast lateral locomotion no overlaps can be detected within contours of consecutive frames (Fig 4b). Similarly, C. elegans moves in a snake like motion so that contour-overlap-based assignments may fail (Fig 4c).

After processing all frames, the overall path of an animal i is given by L i = ( l t 1 i , l t 1 + 1 i , … , l t 2 i ) (2) where t₁ specifies the first frame the animal appears at and t₂ indicates the last valid measurement (1 ≤ t₁ ≤ t₂ ≤ T; T represents the total number of frames).

The initial definition of the larval orientation is based on two regions with the sharpest acute angles. Due to the non-rigid body wall or low per-animal resolutions this assumption is not true in some frames leading to alternating head/tail assignments. Furthermore, other model organisms like C. elegans can imply other curvature characteristics compared to larvae which leads to the necessity to correct the initial h and t calculation.

To adjust the assignments in L i (Eq 2) first distinct sequences L i ˜ ⊆ L i where the animal is not coiled are determined. Afterwards, a probability indicating whether the head/tail assignment is correct or not is calculated for all tuple l t i ∈ L i ˜ based on the following constrains:

If the head and tail are swapped, the spine points s_i and the radii r_i are reversed, too. Furthermore, all spine-point derived features are recalculated. Note that, although these constraints are derived from larval locomotion, the resultant probability is still valid if a C. elegans worm moves forward in more than 50% of the frames. It is worth mentioning that, after identifying h and t, the position of these points along the spine is fixed by assigning each subsequent head/tail point based on the respective predecessor with smallest Euclidean distance. Furthermore, even if the head and tail are swapped one click in the results viewer module is sufficient to correct the model throughout the entire recording (see Results reveiwer module).

To quantify the locomotion in more detail several primary, secondary, and motion-related features are calculated by FIMTrack.

Primary features. In addition to the representation of an animal l t i (Eq 1) the area A and the perimeter P are calculated.

Secondary features. The main body bending angle γ is calculated based on the head h, the mid spine point s L 2, and the tail t. Given the vectors v 1 = h - s L 2 and v 2 = s L 2 - t, Eq (3) is used to calculate the bending in degree. γ = arccos 〈 v 1 v 1 , v 2 v 1 〉 · 180 ° π (3)

As a consequence, an animal is not bended if γ = 180°, bended to the left if γ > 180° and bended to the right if γ < 180° (Fig 2c). Since a user-specified number of spine points can be extracted these points can be used to extract other bendings along the spine by using Eq (3) with appropriate v₁ and v₂ (e.g. to quantify the stereotypical S-shape of C. elegans).

Given a threshold τ_bend, an animal sweeps to the left if γ ≥ 180° + τ_bend and sweeps to the right if γ ≤ 180° − τ_bend. Furthermore, the spine length S_l is calculated by summing up the Euclidean distances between the head, all spine points, and the tail: S l = h - s 1 2 + ∑ i = 1 L - 3 s i - s i + 1 2 + s L - 2 - t 2

In case of coiled animals (Fig 5a) the spine calculation fails to extract the posture correctly. As a consequence, all spine-related features are not reliable. To mark these ambiguous situations, a binary indicator c? is introduced to determine coiled states. The coiled indicator is true if one of the following constraints is satisfied:

Motion-related features. Most of the motion-related features are calculated based on the animal’s center of mass m since this point is calculated directly from the contour and does not depend on the spine calculation. The accumulated distance for an animal at time t is calculated by d acc = ∑ i = 1 t - 1 m i - m i + 1 2 and the distance to origin is given by d org = m 1 - m t 2 Furthermore, the velocity for time t is calculated by v m = ∥ m t - fps 2 - m t + fps 2 ∥ 2 fps where fps is the given frame rate. In a similar fashion, the acceleration is obtained by using consecutive velocities: a m = ∥ v m , t - v m , t + 1 ∥ 2 2

To identify if an animal is in a go phase, a binary indicator g? is used. The following constraints must be valid for a go phase:

To avoid alternating go phase measurements, a user-specified minimal go phase length (τ_go) is used to extract continuous phases. This implies that the number of consecutive g? = true measurements has to be ≥ τ_go to classify this sequence as a go phase. An animal is in a reorientation phase if g? is false.

Stimulus-related features. In order to extend the capabilities of FIMTrack it is possible to place different stimulus markers on the raw image in the results viewer. The following markers are supported:

For all markers the additional features distance to stimulus, bearing angle to stimulus, and is in stimulus region are calculated for each time point and animal. The distance to a stimulus is given by the Euclidean distance between the center of mass m of the animal and the point p representing the nearest point on the stimulus. The bearing angle β is obtained by using Eq (3) with v₁ = m − t and v₂ = p − t (note that the tail t is used since it is not affected by head casts).

Given a point stimulus the necessary computations are straight forward. The nearest point between the animal and a line stimulus is obtained by performing an orthogonal projection of m on the line defined by the line segment. If the projected point p is not located on the line segment we take one of the two endpoints of the line segment as the nearest point which has the minimal Euclidean distance to p. In case of a rectangular stimulus the nearest point p is calculated by p = arg min_pi ∥m − p_i ∥₂, where p_i is the associated orthogonal protection of m onto each of the four boundaries of the stimulus. Since the calculation of the exact nearest point p of m on a ellipsoid stimulus cannot easily be performed in an analytical fashion, we use an approximation of p. Given an axis-aligned ellipse centered at c = (c_x, c_y). First, a line l going through m and c is determined. Next the intersection points p₁ and p₂ between l and the ellipse are obtained and the intersection point with the minimum distance to m is taken as the approximation of p.

A detailed overview of the center of mass progress is given in Fig 8a. Each boxplot represents the center of mass deviation for the respective larva. None of the measurements has a median deviation above 3 pixels. This suggests that the divergence is rather the result of an inaccuracy of the tracking algorithm but more likely caused by the previously performed image processing and definitely influenced by the non-contour-based center of mass extraction in the ground truth data.

The central spine point location contains more outliers compared to the center of mass measurements. The maximal deviation (including outliers) between a measurement and the ground truth is 16.84 pixels (Table 1). As illustrated in Fig 8b measurements for larva 6 include most outliers. These inaccuracies are caused within several frames in which the animal is coiled resulting in an erroneous spine calculation (Fig 8d). The median spine point deviation is below 2 pixels and after removing the outliers the maximum distance decreases to 4.58 pixels (Table 1).

To further study the accuracy an overlay of ground truth and calculated central spine point trajectories is given in Fig 9. Since no sub-pixel accuracy is used for tracking, the calculated path contains more straight lines interrupted by edges. However, the deviation from the ground truth path is rarely more than one pixel.

A high tracking precision can also be observed within the body bending quantification: the mean deviation is below 4° (Table 1) which is depicted in the top left corner of Fig 8c where the head of the larva is given in red, the tail in blue, and the central spine point in black. The mean deviation is indicated by the light yellow area visible at both sides of the spine segment connecting the head and the tail.

By taking a closer look at the plots given in Fig 8c, it can be seen that again only larva 6 includes several frames with a very strong deviation. Since the deviations go up to 171° the head and the central spine point are swapped which can be observed in Fig 8d.