Images of the face were collected, and the images were corrected for ambient lighting changes and device dependence using a per-image calibration method. A standard way to do this is to include a color chart in each image [7, 8], for example the X-Rite ColorChecker Classic. A schematic of the method is shown in Fig 1. A 3 × 3 color mapping from device RGB to the standard device-independent XYZ space can then be developed using the chart, and applied to data from that image alone. Even when under relatively spatially homogeneous ambient lighting, there is likely to be a variation in the illumination intensity across the chart. Finlayson et al proposed using an iterative alternating least squares (ALS) method—the shading across the chart is estimated keeping the color mapping fixed, then the color mapping is estimated using the shading estimate, and the process repeated until convergence on M [9]. The median RGB value for the forehead skin ROI (carefully chosen to avoid specular reflection) from single ambient images were extracted and the per-image color mapping applied to move to XYZ space, [ XYZ ] = [ RGB ] M (1) The resulting XYZ values are independent of the device and ambient conditions. The final stage is to link these color values to bilirubin levels. Chromaticity values are used to avoid issues caused by shading, different exposure times or phone-eye distances across images. Chromaticity is defined as a color channel divided by the sum of the other channels—for example x chromaticity is given by x = X/(X + Y + Z). The prediction of bilirubin is based on a regression model established between TSB and x and y chromaticities of all patients [10]—i.e. predicted bilirubin = p × x + q × y + r, where x and y were chromaticities calculated from the RGB pixel values of the region of interest, and p, q and r were parameters estimated from a linear regression model with x and y being the independent variables and the measured bilirubin (TSB) being the dependent variable. All samples in the dataset were used to optimise the estimation model, which was then applied to predict bilirubin in all patients. To assess the performance of the estimation model, the correlation between TSB and the predicted bilirubin was calculated and used as the main result.

An alternative processing method was used for images captured of the sclera and lower eyelid, as described in [6]. This approach avoids the need to include a color chart in each image. The most relevant aspects and improvements for this application are summarised here. An overview of the process is shown in Fig 2. Flash/ no-flash image pairs are captured, and data from the no-flash image subtracted from the flash image. For a given device, the effective illumination is thereby standardised to the flash of the phone alone [6, 10]. The subtracted signal to noise ratio (SSNR) can be used to ensure that there is sufficient difference between flash and no-flash images in the region of interest [6].

When approaching scleral imaging in adults, there are typically many blood vessels and regions of specular reflection which can distort results. We therefore use a simple method to remove the majority of these pixels from the selected region of interest. A test for specular pixels is first carried out based on the presence of very high channel values within the region of interest. If specular pixels are deemed present, a modified and simplified version of [11] is used to remove these: the image is converted to the Hue Saturation Value (HSV) color space, a power law enhancement applied to the V channel and Otsu thresholding carried out on the result [12]. Vessel filtering is then carried out on the resulting pixels in the region of interest by calculating the red chromaticity of the image and using Otsu thresholding. The median can then be calculated for the remaining pixels in the flash and no-flash images, to remove any remaining outliers and minimize the effects of movement artefacts, before finally applying subtraction.

Once the results are standardised for a given phone, the second step is to remove dependence on the device—even variations between handsets of the same model will impact accuracy [13]. Images of a color chart with patches of known colors, such as the X-Rite ColorChecker Classic, are captured under the phone’s flash illumination with no ambient light. A color mapping M can be developed once and applied to all subtracted data for a given phone, since the effective lighting is standardised to the flash of the phone [6], moving to XYZ space as in (1). It is very important to capture and account for the high level of intensity non-uniformity of the flash across the color chart in order to maintain a high color mapping accuracy [6]. For this reason, the ALS approach described above was used to develop the per-device Ms [9]. Using this approach, it is possible to avoid the need for a phone tripod and grey card, typically used to measure the shading field. Instead, the one-time calibration image capture process is simplified through the use of just handheld images of the color chart. As described in the previous section, predicted bilirubin levels can then be generated based on the xy chromaticities [10].

One hundred and three sets of images of patients with cirrhosis were collected at the Royal Free Hospital in London. Ethics for this study was granted as part of an on-going biomarker study approved by the institutional review board (London—Harrow; REC Ref: 08/H0714/8) and all patients provided informed consent. Three types of image pairs were collected. The first focussed on the sclera alone, the second focussed on the lower eyelid, and the third focussed on the forehead alongside an X-Rite ColorChecker Classic chart, which (the same one) was used consistently in all patients.

Images were collected in parallel using two phones—an LG Nexus 5X (Nexus) and a Samsung S8 (S8). The Nexus used the screen turned on white as the “flash” with the front-facing camera. This photo capture mode would be useful for patients conducting self-monitoring. The S8 used the rear-facing camera with the camera flash, where a custom diffuser was used to spread and dim the light (depicted in Fig 2). A custom-developed app was used for image capture on both phones; when opened, the flash turned on to allow acclimatisation, and when capture was requested, a raw flash and no-flash image were saved in quick succession. The total serum bilirubin (TSB) level was also collected via a blood test as part of standard care. Ideally all three image types would have been collected using both phones for each patient image set but this was not always possible in practice. The different numbers of available images for each category of images are reflected in the presented results.

To avoid repeating the time-consuming process of capturing clinical data, it is vital that image data is collected in such a way that it can be processed to remove the dependence on both the ambient conditions and the capture device. Here, we demonstrate the importance of fully applying processing for the two methods used in this paper before focussing on comparing the results from different imaging sites for a single phone.

Fig 3 presents forehead data for the S8 phone before and after applying the per-image mappings. Without attempting to correct for ambient lighting (or device dependence), no trend is seen. In this case, it is very clear that the full processing is necessary—when using the per-image mappings there is a strong significant correlation observed. Further detail and discussion on the data itself is presented in the next section.

Fig 4 demonstrates the importance of accounting for device dependence when using the two-step method outlined in the second image capture methods section. Two models converting sclera color to predicted TSB were developed. The first was developed after applying ambient subtraction, thus standardising the lighting, but without applying the device-specific color mapping. This situation often occurs in pilot studies, where care is not taken to remove device dependence. The second model was developed after applying the color mapping, whereupon data should be independent of the lighting and device used for image capture. The results from the S8 are shown in blue in Fig 4. The difference in accuracy with and without the color mapping is negligible for this phone.

We then consider the situation where further data is collected using a different phone—the Nexus. If the first model is used, predicted TSB levels for the Nexus are extremely different from both the S8 and ground truth TSB. This is highlighted by the large number of negative predicted TSB values shown in red in Fig 4(a), and the strong trend observed in the Bland Altman plot in (b). Without the one-time calibration step, an entirely new set of patient data would need to be collected using the Nexus phone (or any other new device) in order to obtain reliable predictions. However, when device dependence is accounted for both in model development and Nexus processing, the mean squared error of the predicted bilirubin compared to TSB is no longer significantly different between the two phones (t-test, p = 0.34, df = 71). This is demonstrated in Fig 4(c) and 4(d), noting the 50% reduction in the 95% confidence intervals in (d) of -98 to 96μMol/L compared to (b) with -218 to 231μMol/L. When the method of [6] is followed, it is, therefore, possible to collect just one set of paired phone images and TSB values to develop a model, and then use this model to obtain reasonable predicted values using a different phone, and, in the case of the Nexus phone, a different camera (front) and illumination (white screen).

Having validated that the processing used accounts for variations introduced by ambient light and different devices, we focus on comparing results from different imaging sites. Data presented here is from a single phone S8 for clarity. Figs 5, 6, 7 and 8 present predicted bilirubin as a function of measured bilirubin and associated Bland Altman plots for models based on data from images of the forehead, sclera, lower eyelid and sclera and lower eyelid combined, respectively. The correlation of predicted and measured bilirubin, and the limits of agreement of the Bland Altman plots are used to assess the quality of the results.

The results from Fig 5 using the forehead are better than anticipated based on previous work [3, 14]—since no correction for melanin was performed it was expected that the performance would be affected. However, the input images were highly biased, with 80% of Caucasian patients, 15% of Asian patients and just 5% of Black patients. This bias likely explains the good performance observed with this dataset, and these results alone should not be taken as an endorsement of extracting color from the skin without attempting to correct for melanin.

The best results using a single imaging site are yielded from the sclera, as shown in Fig 6. The lack of melanin in the sclera will have helped, however the reduced size and reflective nature were challenges. Patients were largely able to comply with instructions to look in a particular direction to expose more of the sclera. Additionally, applying the filtering approach detailed in the Method section removed the majority of pixels affected by reflection and containing blood vessels.

Results based on the lower eyelid, shown in Fig 7, are similar to those from the sclera. In fact, the mean squared error of the predicted bilirubin compared to TSB is not significantly different when using the sclera or lower eyelid as the ROI (t-test, p = 0.18, df = 65). This is a surprising result as the change in color to the naked eye is far less pronounced in the lower eyelid than the sclera. Image capture for the lower eyelid dataset was more challenging than for the sclera, since the process of sufficiently pulling down the eyelid was less comfortable for patients and more prone to error than simply looking to one side. Given their similar results and the easier image capture, the sclera is clearly the better imaging site for this application.

Rather than generating a model for bilirubin based on just one imaging site, a multivariate approach is possible [15]. Fig 8 presents results generated by combining data from the sclera and lower eyelid. Whilst currently images are captured separately for the sclera and lower eyelid to maximise the quality of both, it would be possible to extract the scleral color from one single lower eyelid image. However, as discussed above, for clinical translation and independent image capture, use of the lower eyelid would have low patient acceptability.