Eight rural hospitals (H1 to H8) were chosen purposefully from four of Kenya's eight provinces to provide some representation of the variety of rural hospital settings encountered in Kenya (Table 1) [12]. Hospitals admitting a minimum of 1,000 children and conducting at least 1,200 deliveries per year were eligible for inclusion. Prior to the study, medical records documenting admission information were written as nonstandard, free-text notes in all eight hospitals. The Ministry of Health usually aims to disseminate national guidelines aimed at hospital care to facilities through distribution of some print materials and ad hoc or opportunistic workshops or seminars. It had not previously been able to augment this approach with systematic efforts or provide specific supervision to support paediatric hospital care. Further, none of the eight hospitals themselves had explicit procedures for implementing new clinical guidelines.

We collected data from medical records of paediatric admissions aged 2–59 mo to describe paediatric care practices of clinicians and nursing staff targeted by the guidelines, training, and feedback. The Kenya Medical Research Institute National Ethics and Scientific review committees approved the study (Texts S1 and S2).

Prior to inclusion in the study the eight shortlisted hospitals were visited and meetings were held with the hospital management team. At these meetings, the study design, randomization, potential inputs, approach to data collection, and longevity were explained. All hospital management teams subsequently assented to their hospital's participation and randomization after internal discussions. Assent from the hospital's catchment population was not sought. Staff in all hospitals were made aware of the study's overall aims to explore ways to improve care and need for data collection through specific presentations made after randomization at the start of introductory training and using written information sheets. After obtaining the hospitals' assent we allocated eight hospitals (clusters) to a full (intervention group, hospitals H1–H4) or partial (control group, hospitals H5–H8) package of interventions using restricted randomization. Of 70 possible allocations, seven defined two relatively balanced groups (Table 1). These allocations were written on identical pieces of paper, with hospitals represented by codes, and one allocation was randomly selected using a “blind draw” procedure. Participating hospitals and the research team could not be masked to group allocation. However, information on group allocation was not publicly disseminated and the geographic distance between hospitals was large. We therefore do not feel that users of the hospitals were aware of or influenced by the form of intervention allocated to the hospital.

The intervention delivered over 18 mo (from September 2006 to April 2008) aimed to improve paediatric admission care by promoting hospitals' implementation of best-practice guidelines and local efforts to tackle local organizational constraints. Before the trial commenced, a decision was made to adjust the timing of the primary endpoint for measuring intervention effectiveness, aligning it with the end of this 18-mo active intervention period. As part of this updated approach, monitoring of intervention sites was planned to continue for 12 mo after active intervention had ended. Funds were not available to support comparable extended monitoring in control sites. The intervention components are labeled 1–6 and a–c in Figure 1 [21] and included: (1) setting up a scheme for regular hospital assessment through surveys conducted six monthly, followed by (2) face-to-face feedback of findings in intervention sites, and (a) written feedback in both groups. The other components were: (3) 5.5-d training aimed at 32 health workers of all cadres approximately 6–10 wk after baseline surveys (July to August 2006) in intervention hospitals [13], (b) provision of clinical practice guidelines introduced with training, (c) job aides, (4) an external supervisory process, and (5) identification of a full-time local facilitator (a nurse or diploma-level clinician) responsible for promoting guideline use and on-site problem solving [19]. Supervision visits were approximately two to three monthly, but facilitation remained in place throughout the 18 mo. The package for control sites (H5–H8) included five components (1, 6, a, b, and c): (1) six-monthly surveys with written feedback only, provision of (b) clinical practice guidelines and (c) job aides, and (6) a 1.5-d initial guideline seminar for approximately 40 hospital staff. The design thus compares two alternative intensities of intervention, both providing considerably more than routinely delivered, although we refer to one arm as the “control.”

One of the job aides, introduced to all sites with all training and continuously supplied to improve documentation of illness, was a paediatric admission record (PAR) form. This was to replace traditional “blank paper” medical notes [22]. All hospitals were aware that their records and patient management were to be regularly evaluated. All job aides, training materials, and assessment tools are available online (http://www.idoc-africa.org/docs/list/cat/5/subcat/27).

Data were collected at baseline and then at six-monthly intervals during six and four surveys in intervention (surveys 1–6) and control hospitals (surveys 1–4), respectively (Figure 1). A single survey took approximately 2 wk with all sites surveyed within a maximum 6-wk consecutive period by employing up to four teams. The survey tools and team training have been described in detail elsewhere [14]. In brief, data were collected using three tools adapted from previous work [7],[8] then extensively pretested: a checklist of structure indicators, patient case-record data abstraction forms, and a structured parent/guardian interview tool. In the case of the parent/guardian interview formal, written consent was obtained prior to data collection with no parent/guardians refusing consent. Ethical approval was granted for confidential abstraction of data from archived case records without individuals' consent. Survey team leaders remained the same throughout the study and teams received 3 wk initial training that included a pilot survey. Data collectors could not be blinded to allocation, but all were guided by standard operating procedures and, for case records, a 10% sample were independently reevaluated by the survey supervisor during each survey. Agreement rates for data abstracted were consistently greater than 95%.

Case records from a random sample of calendar dates from the 6-mo intersurvey periods were selected with the proportion of dates sampled adjusted to yield approximately 400 records based on hospitals' admission rates. On the basis of prior experience we aimed to conduct interviews with 50 caretakers of admitted children during each 2 wk survey (surveys 1–4).

Primary effectiveness measures were 14 process indicators measured on paediatric admissions aged 2–59 mo at 18-mo post baseline (survey 4). Secondary measures were four valued system outcomes of admission and changes in structure measured at the hospital level. The trial was not designed to evaluate mortality effects.

There were 70 district hospitals in Kenya at the time of the study. Hospitals from four of Kenya's eight provinces without potentially confounding, nonstudy interventions and meeting the outlined eligibility criteria were shortlisted. Data on additional criteria felt to help define the range of contexts in Kenya were then evaluated, and eight hospitals from four provinces were purposefully selected to ensure that at least two out of these eight hospitals met each positive and negative criterion (Table 1), that two hospitals were from each of the four provinces and represented logistical implications of their location. The sample size of eight hospitals was estimated using two approaches to compare performance within each hospital (plausibility design) and across the two arms of the trial (cluster RCT analysis). Within hospitals, we estimated that 50% correct performance could be estimated with precision (95% confidence intervals [CIs]) of ±7% with 200 admission records (50% of 400 sampled admissions), or, ±10% with 100 admission records. The second calculation for group (C-RCT) comparisons accounted for the clustered nature of the data. The median intraclass correlation coefficient (ICC) for 46 quality of care variables estimated from a health facility cluster survey in Benin was ρ = 0.2 [24]. We estimated, employing this value for the ICC, that 100 observations per cluster would provide 80% power to detect a 50% or greater difference in proportions between intervention and control arms at 18 mo follow-up [25].

Data were double entered and verified in Microsoft Access and analysed using Stata, version 10 (Stata Corp.) according to the prespecified analysis plan.

Results were similar from both approaches used to compare intervention arms, i.e., adjusted comparison at 18 mo and difference of differences. For brevity, we outline only the results of adjusted comparisons and present other data in Tables S3, S4, S5.

Our study has limitations. Hospitals were not selected at random from a set of all eligible hospitals for logistic reasons and, because random selection of a small number of clusters may not have produced balance nor guaranteed representativeness at baseline. Hospitals assented to participation and randomization, but we were not able to engage communities in this process [44], and they and survey teams were aware of intervention allocation. The latter is a potential problem with results based largely on retrospective review of records. The discrepancy between documentation and performance presents a particular threat at baseline before efforts in all sites to improve clinical notes. Prescription data are less susceptible to this limitation however, and improved prescribing paralleled improvement in assessment indicators. Efforts to minimize possible observation bias at the point of data collection included the use of structured inventory forms, standard operating procedures, and extensive training in survey methods. With only four hospitals per group, attempts to adjust for baseline imbalance may also have only limited success. However, to facilitate scrutiny we report on the context of intervention [19],[20], its delivery and adequacy [12], the views of intervention recipients [18], and detailed site-specific data (see Tables S1, S2, S3, S4, S5) and suggest that all are considered for a complete interpretation of this study of a complex intervention.

Demonstrations that a similar intervention package is effective in other settings would strengthen the evidence supporting widespread adoption. While there are few studies of this nature reported, we note the recently reported success of multifaceted interventions in middle- and high-income countries [35],[45]. However, standardizing complex interventions may be difficult, if not impossible, given the important role of context in shaping mechanisms and outcomes [46]. For this reason, future reports will attempt to provide detailed insight into how and why this intervention met with general but varying degrees of success. If our results are deemed credible, however, the data we present have a number of implications. Firstly, current efforts to implement and scale up improved referral care in low-income settings need to go beyond the existing tradition of producing and disseminating printed materials even when linked to training [15]. Instead broader health system efforts, guided by current understanding of local contexts and capabilities and theories of change, are required.

Within Kenya it would obviously be a mistake to consider that the intervention package tested can be scaled up simply by aiming for much broader coverage with the training course we designed. Effectiveness has been demonstrated only for the multifaceted intervention. Thus, scaling up should aim to provide all inputs not just guidelines, job aides, and introductory training. However, providing regular support supervision and performance feedback related to child and newborn care at first referral level are not routine. Resources and systems for supervision need strengthening and supervisors themselves will need training and organizing. Routine information systems are inadequate to generate the data required to evaluate care, and capacity for conducting and disseminating analyses as part of routine feedback is largely absent. The role of facilitators is also not one that currently exists. Although the roles required could perhaps be played by senior departmental staff, the lack of human resources means such tasks cannot simply be added to already busy jobs [19]. Furthermore the skills or desire to facilitate change are not necessarily present amongst such mid-level managers.

Countries other than Kenya considering adopting the approach may have similar limitations. In addition they may need to tailor some intervention components to their particular setting. For example, the detail of a clinical guideline or job aide or approach to training may need to reflect available resources or local evidence. However, such adaptation would need to be complemented by careful consideration of how systems can be made ready to support implementation of new practices and improved quality of care. We would suggest this includes due attention to influencing the institutional culture and context of rural hospitals although willingness to invest in more integrated approaches often seems lacking [47]. Finally, before making decisions on implementation, policy makers increasingly require carefully collected and reported cost-effectiveness data. Such a report is in preparation. Considering only the financial costs of specific inputs, for example the typical 5-d training course for 32 participants at approximately US$5,000 [13] or the annual cost of a facilitator at less than US$5,000 [18], while of some value, are insufficient for prioritizing resource use.

Our findings provide strong evidence that a multifaceted intervention can improve use of guidelines and, more generally, the quality of paediatric care. Cost data will help determine whether this implementation model warrants wider consideration as one approach to strengthening health systems in low-income settings.