The KeepCoool study was conducted in German general practices from a university teaching practice network (n = 185) from February to October 2018. Typically, these physician-owned practices are one to three physician practices with about 2 practice assistants per physician with larger group practices currently emerging. The practices serve patients covered by the statutory health insurance (about 90% of the population) and private insurances (about 10% of the population). Practices frequently have long lasting relationships with their patients.

Recruitment followed a structured approach: contact by phone and fax up to three times or until practices responded. A trained health services researcher (A.T.) and a trained study assistant visited all practices which had volunteered for participation. They had been involved in other practice studies and were familiar with the specifics of the setting as well as the topic. They informed the practice teams about the study and the study materials, obtained written informed consent, conducted the baseline survey, and placed the data logger in the refrigerators. Details are published [25, 26, 28].

The study was originally designed as a randomized, controlled trial with intervention arm (study arm 1) and waiting-list control arm (study arm 2) in practices with confirmed cold chain breaches [28]. In addition, the original design included a third arm to follow practices with temperatures in the target range (study arm 3: monitoring only). However, temperature measurements of the first 7 days (baseline) were far worse than expected: 68% refrigerators were outside the target range, and 15% had reached critically low temperatures <0°C for up to 7 days [25]. Subsequent ethical considerations motivated the change to a study with pre-post-design to avoid the use of vaccines with reduced potency and to assure patient safety. Thus, the study arms 1 and 2 were combined resulting in a total of two study arms: intervention arm (A) with practices having cold chain breaches at baseline. The monitoring only arm (B) with practices without cold chain breaches at baseline remained identical (former study arm 3). Thus, access to the eLearning program KeepCoool was offered to all practices with cold chain breaches at baseline (intervention arm A). For analyses, the practices in the intervention arm A were later grouped according to their participation or non-participation in the eLearning. We aimed to investigate if the participation in the eLearning improved refrigerator temperatures. Data logging remained identical for all practices. Temperature data logging was continued every minute until the follow-up practice visits after a minimum of five weeks.

For analysis, refrigerators served as their own control. Separately, refrigerators in the target range at baseline were monitored to evaluate if temperatures were maintained (monitoring only). The responsible ethics´ committee was informed on the protocol change. All practice sites originally included remained in the study.

The best practices-based eLearning KeepCoool provides learning content in five tutorials: temperature, refrigerator, storage, responsibilities, and monitoring. When starting, participants completed an 11-item knowledge questionnaire as basis for tailored learning. The eLearning used three didactic elements: a) personal address, b) targeting of the respective profession (physicians or practice assistants), c) individualized feedback tailored to a participant’s baseline knowledge. The program followed a consistent presentation with `basic information´, `tips for the practice´, and `expert knowledge´ for in-depth understanding. Options to download relevant literature and practice templates were available. After the eLearning, participants who successfully answered ≥ 7 of the same 11 questions (> 60% correct answers as required by the regional physician association) received a certificate. For details [27].

Temperatures were measured using a data logger (testo 175T) with an accuracy of ±0.4°C in the operating range -5°C to +10°C (calibrated under a DIN EN ISO 9001:2008 certified quality assurance system). The device was equipped with a standard probe which measured ambient air temperatures inside the refrigerator at 1-minute intervals similar to prior studies [10, 12, 15, 29, 30]. We abstained from glycol probes [28] due to limited resources and an easier handling of the new device, which remained inside the refrigerators throughout the study with the display turned off and memory access locked [25]. For the analyses, the first 120 minutes after each data logger set-up were excluded to allow for probe acclimatisation.

Using data from all practices, the effectiveness of the intervention was determined using 7-day-temperature readings (10,080 recordings) of the 1^st week (baseline period) compared to the 6^th week (follow-up period). The primary outcome was the percentage of refrigerators with temperatures within the target range (2 to 8°C) in the follow-up period. Secondary outcomes comprised various parameters, e.g., the prevalence of freezing temperatures (see Table 2), structural changes of refrigerators, and a Euclidian temperature prediction model. For the three temperature zones (+2 to +8C; ≤0°C; ≤ -0.5°C) we calculated the hours in the respective zone (so called: cumulative time) and the longest period in the respective zone (so called: longest consecutive time). In addition, we noted if—and when—the practice assistants and the physicians finished the KeepCoool eLearning knowledge test and received the certificate.

The study complies with the ethical principles of the World Medical Association Declaration of Helsinki. Ethical approval was obtained from the Ethic Commission of the Medical Faculty of the University of Duisburg-Essen (reference number: 14-6118-BO). Participants provided written informed consent. The study was registered in the German Trial Register: DRKS00006561 (date of registration: 20 February 2015).

Of the 168 practices contacted, 64 participated (response rate: 38.1%) with 75 refrigerators. At baseline, 73.4% of the practices (47 of 64) showed cold chain breaches in ≥ 1 refrigerator and were offered the eLearning. This offer was accepted by 48.9% of the practices (n = 23 of 47) (`participating practices´), while 24 practices did not participate (`non-participating practices´). Follow-up data were available for 44 of 47 practices (one drop-out among participating practices, two among non-participating ones). As separate third group, practices without cold chain breaches at baseline were followed (monitoring only practices). The subpopulations were comparable for key characteristics (Table 1 and Fig 1).

Overall, we observed improvements in refrigerators and temperatures, most of which were significantly better in the participants than the non-participants, although these practices did not reach the quality of the monitoring only group (Table 2).

The primary outcome (continuous temperatures within target range of 2 to 8°C at follow-up) was reached by 4 of 24 (+16.7%) refrigerators. Improvement was noted also for secondary outcomes: critically low temperatures (≤0°C) 62.5% less at follow-up; in the target range in 95% of the time while always >0°C improved by 20.8% (baseline: 16.7%; follow-up: 37.5%).

Among participants, one practice had a new refrigerator (unsuitable at baseline) and one had ordered two refrigerators (both unsuitable at baseline). In the non-participants, one refrigerator was defrosted, one practice exchanged their refrigerator with another existing one, one bought a new refrigerator (unsuitable at baseline).

As published prior, the average knowledge score at baseline was 5.6 (SD ±1.9) of 11 correct answers with slight differences between physicians (5.9; SD ±2.3) and practice assistants (5.5; SD ±1.7) [27]. Average knowledge improved to 9.8 ± 1.2 with an increase of 4.2 points (p<0.001). In physicians, the mean knowledge score at follow-up increased by 4.6 points in physicians (p<0.001) and by 4.0 points in practice assistants (p < 0.001) [27].

On average, the first person per practice finished the test 7.4 days (SD 9.16, range 0 to 40, median 5) after receiving the login information. Within practices the duration between the first and last participant who finished the test was 7.9 days (SD 11.93, range: 0 to 40, median 2). Seven practices finished eLearning during the sixth week follow-up or later. For details see Fig 2.

Fig 3 shows the average Euclidean distance of the 95% prediction intervals (considering both, model uncertainty and response uncertainty) from the two points (5, 5) up to and after the first participant finished the eLearning for the participating practices of the intervention group. These two points represent the average value of the interval between 2 and 8 degrees centigrade that is considered a good range for storage of medical vaccines. The smaller the Euclidean distances, the better are the circumstances for medical storage. Euclidean distances were significantly reduced after the first participant finished, as indicated by a paired t-test (p-value = 0.0046).

The model shows if a practice’s participation in the eLearning changed/improved temperature over time given the covariates observed time, weekend status, participation of a physician, number of participating practice assistants and single/group practice. Refrigerators of participating practices in the intervention group were analyzed using N = 2,110,282 single measurements (mean per practice 95,921.91, SD 22,445.52). All p-values were significant after multiple comparison adjustment regarding the false discovery rate [36] level of 0.05 except that weekend measurement did not influence the scale of the distribution. The intervention therefore affected both location and scale of the refrigerator temperature.

The content of KeepCoool is based on best-practices and guidelines from several countries. The eLearning addressed all practice personnel. A selection bias cannot be excluded as practices participating in the elearning may differ from those who did not. Contamination between practices was unlikely as they are spread in a larger area and work independently, but cannot be fully excluded. Some practices participated in the eLearning beyond the period defined for the primary outcome. We addressed this in the Euclidian model with 2 million temperature data from up to 4.5 months. For the effectiveness, we chose a strict primary outcome according to recommendations for vaccine storage, yet slightly less strict outcomes might be as appropriate for modern vaccines (e.g., in target range 95% of the time while always above the freeze threshold). Data on changes of refrigerators or practice personnel were not obtained to not burden busy practice personnel. However, we assured that only personnel who had participated in the baseline evaluation were provided access to the elearning. The Euclidian distance used the +5°C median as guidelines recommend to target 5°C when adjusting refrigerators. No long-term monitoring, e.g. after 6 to 12 months, and room temperature monitoring were performed but recommended for future studies.

Overall, the eLearning KeepCoool showed high learning effectiveness [27] and significant improvements regarding the vaccine cold chain. The freely available eLearning allows for the nationwide scaling up to all personnel and practices vaccinating: https://keepcoool.ukbonn.de/.