QOF+ was launched by Hammersmith and Fulham primary care trust in West London during September 2008. The primary care trust serves around 180 000 residents covered by 31 general practices and has two main acute hospitals. The resident population is young (one third aged 20–34 years), mobile (12% turnover a year), and culturally diverse (22% from ethnic minorities) with considerable income inequality.

Annual patient-level data on all adult patients (≥ 18 years) registered at 31 Hammersmith and Fulham general practices during financial years 2004/05 to 2010/11 were extracted from electronic medical records. The de-anonymized and de-identified extract includes anonymised information on patient demographics, clinical diagnoses and clinical measurements [15]. As the patient-level data does not contain identifiers or patient sensitive information, individual patient consent was not required. Publicly available annual practice-level data of QOF performance for all practices in England for the years 2006/07 to 2010/11 was obtained from the NHS Information Centre. The dataset does not contain patient-level data. Ethics approval was granted by London Queen Square Research Ethics Committee. As QOF+ was introduced in December 2008, we dropped data for the 2008/09 financial year from both the patient and practice level datasets.

Our main outcome measures were mean values and achievement of clinical targets for blood pressure, total cholesterol and HbA1c. For multiple measurements within a period, the last measurement was used for compatibility with the calculation of the performance indicators. Covariates in our patient level analyses included age, gender, ethnicity, body mass index, number of cardiovascular comorbidities and area socio-economic status (based on the index of multiple deprivation 2007) [16]. Age was divided into three categories: 18 to 44, 45 to 64 and 65+ years. Body mass index was split into the three categories below 25, between 25 and 30 and above 30 kg/m². Ethnicity is categorized into White, Black (Caribbean and African), South Asian (Indian, Pakistani and Bangladeshi) and Other (including Chinese). Based on indicators for coronary heart disease, diabetes, hypertension, stroke or transient ischemic attack, atrial fibrillation and heart failure, we calculated the number of cardiovascular comorbidities (0,1,2+) per patient. Annual data on whether a patient had been exception reported from QOF/QOF+ was obtained.

Patient records with diastolic blood pressure not inside the interval 20 and 160 (excluding the limits) were discarded. Similarly, records with systolic blood pressure not inside the interval 30 to 250 (excluding the limits) were discarded. Records with cholesterol values greater or equal to 15 were removed as well as HBA1c values greater or equal to 20. Body mass index has an additional category for missing values, indices below 10 and above 70 due to the high number of records with missing values. Note that missing values cannot be imputed by interpolation because all patients with at least one missing value for body mass index have no body mass index given in any year.

As there were changes in the business rules for QOF over the study period we applied the same version (version 16 rule set) across all years.

We conducted three different analyses at an a-priori chosen significance level of 5%. The first analysis tested whether QOF+ was associated with improvements in target achievement relative to national trends. The second analysis tested whether QOF+ was associated with an increase in exception reporting and whether any changes in exception reporting influenced target achievement. The third analysis investigated the impact of QOF+ on actual clinical values.

Table 3 shows the results of the difference-in-difference approach. In the first column, we observe that all but four indicators (CHD8, DM17, STROKE8 and COPD8) treatment and control practices have a similar pre-intervention trend as required by a difference-in-difference approach based on a conservative significance level of 10%. Fig. 1 visualizes the trend for BP5 (incentivised within QOF+) and COPD10 (not incentivised within QOF+). The effect of interest is the differential impact of QOF+ on treatment practices in the second column of Table 3. All QOF+ stretched indicators have a significant and positive coefficient. For example, BP5 target achievement rates increase by additional 3.7% points for QOF+ practices (p-value <0.001). At the same time, the four control indicators for COPD and CHF, which are not subject to additional incentives in QOF+, do not show significant differential effects between QOF+ and control practices. As an example, COPD10 target achievement rates do not differ significantly between QOF+ and control practices (p-value 0.176). Thus, for all stretched indicators, which follow a parallel pre-intervention trend, the target achievement rates significantly increased upon introduction of QOF+; while target achievement rates of unmodified indicators were not affected.

Changes in target achievement can either be caused by changes in the number of exception reported patients or by an increase in the number of well-controlled patients. We can disentangle these effects by analysing changes in the number of exception reported patients and controlled patients. First, we estimate the changes in the proportion of exception reported patients relative to all patients. Table 4 shows the baseline from 2005 and the secular annual change of exception reported patients in the first two columns. Only the indicator BP5 has a significant change in exception reporting over time, decreasing by 1.3% points (17% relative to the baseline 2005) per year (p-value <0.001). This decreases the rate of exception reporting for BP5 from over 8% in 2005 to 4% in 2008 before the introduction of QOF+.

With the introduction of QOF+, the fraction of exception reported patients increases significantly for five indicators (third column in Table 4); BP5 (5.3% points, p-value <0.001), CHD6 (2.4% points, p-value 0.028), CHD8 (3.7% points, p-value 0.029), DM24 (6% points, p-value 0.018) and DM25 (4.3% points, p-value 0.049) between 2 and 6 percentage points. The QOF+ effect represents the average change from before 2009 to after 2009 with consideration of the secular trend. For example, BP5 would have further decreased by 1.3% points per year without the introduction of QOF+. However, the introduction of QOF+ increased the rate of exception reporting in average for 2010 and 2011 by 5.3% points relative to the expected secular decrease.

Table 4 shows the impact of QOF+ on the proportion of controlled patients relative to all non-exception reported patients in the last three columns. All indicators but STROKE6 show a significant secular improvement in the proportion of controlled patients ranging from 1.1% points (CHD6, p-value 0.016) to 4.2% points (STROKE8, p-value 0.001). For these indicators, there is no additional significant change of the proportion of controlled patients upon the introduction of QOF+ (p-values ≥ 0.278). Only STROKE6 has no significant secular trend (p-value 0.693) but the proportion of controlled patients significantly improved upon introduction of QOF+ by an average of 9.7% points (p-value 0.04) in 2010 and 2011.

The impact of QOF+ on clinical outcomes is shown in Table 5 subdivided for different groups of patients with an illustration in Fig. 2. The introduction of QOF+ was associated with a statistically significant increase in blood pressure in patients with hypertension (diastolic 0.54 mm Hg, p-value <0.001; systolic 1.81mm Hg, p-value <0.001) and diabetes (diastolic 0.80mm Hg, p-value 0.002; systolic 1.70mm Hg, p-value <0.001) and an increase in systolic blood pressure in patients with stroke (1.90mm Hg, p-value 0.026). For exception reported patients, only the diastolic blood pressure worsens significantly for stroke patients (6.92mm Hg, p-value 0.005). For non-exception reported patients, the blood pressure deteriorates significantly for diabetes patients (diastolic 0.64mm Hg, p-value 0.017; systolic 1.50mm Hg, p-value 0.001) as well as the systolic blood pressure for hypertension patients (1.34mm Hg, p-value <0.001). However, the cholesterol value improves significantly for coronary heart disease (-0.07mmol/l, p-value 0.013).

The introduction of local pay for performance programme (QOF+) had a significant impact on target achievement for quality indicators subject to enhanced financial incentives. Most of the improvements in target achievement were due to increases in exception reporting. The programme was not associated with discernable improvements in overall clinical quality.

The effect of setting more demanding targets within existing pay for performance programmes on clinical performance has been little investigated. One previous study examined the impact of an increase in the upper payment threshold for influenza immunization in CHD patients from 85% to 90% in QOF during 2006/07 [17]. The findings suggest that this change was associated with modest increases in the proportion of CHD patients immunised (0.41%, CI: 0.25–0.56%) but that the proportion exception reported (0.26%, CI: 0.12–0.40%) for this indicator also increased. Our study builds on this previous study by demonstrating that a local QOF+ programme, which set more ambitious payment targets, was associated with increased exception reporting across different clinical outcomes and more disease groups.

Our study benefits from several strengths in the design of the analysis. Unlike other pay for performance studies, our models take the underlying secular trends into account using a time-series approach [18]. In addition, we adjust for important covariates. In contrast to an evaluation of the national QOF program, we focus on a local pay for performance scheme, which allows us to compare indicators with non-intervention sites using the more robust difference-in-difference approach. In addition, the increased rate of exception reporting cannot be explained by a change of patient composition since the analysis was conducted for patients registered in all years.

Except for the results of the difference-in-difference approach (Analysis 1), our findings may be influenced by other reforms occurring at the same time of QOF+. However, we are not aware of any major quality improvement programmes introduced for CHD, hypertension or stroke care at the time that QOF+ was implemented.

In the analysis of clinical outcomes, we observed that at a population level, QOF+ may have had limited or even negative impacts on risk factor control. This reflects findings from national and local data which suggest that secular improvements in clinical outcomes before the introduction of QOF+ had started to stagnate [5,7]. Without a control group for clinical outcomes not affected by QOF+ (data unavailable), we cannot disentangle a secular stagnation effect from the impact of QOF+. On the other hand, the deterioration can be due to neglect of patients who are exception reported or other subgroups.

Improvements in target achievement associated with the introduction of a local pay for performance programme with more ambitious payment targets than those set in national QOF were mainly attributed to increased exception reporting by practices. Exception-reported patients are less likely to achieve clinical targets [19] and the impact of their exclusion is to increase the cost to the scheme of each patient who does meet the targets [20]. Therefore, implementation of pay-for-performance programmes should be accompanied by measures to prevent higher exception reporting. This requires defining an appropriate level of exception reporting, which is notoriously difficult to assess; or an active monitoring program, which contributes to the overhead costs. More generally, the study suggests a trade-off between additional incentives for better care and monitoring costs, which should be considered already in the design of the program.