The study utilised a Solomon three group randomised controlled trial design [27], with one intervention and two control groups to control separately for intervention and for measurement effects. A more conventional Solomon four group design was not possible given that the intervention required baseline information in order to personalise feedback to each participant. The International Standard Randomised Controlled Trial Number for this study is: ISRCTN30784467. The intervention group received brief personalised normative feedback (PNF) within a few weeks of completing the baseline assessment, which comprised an online questionnaire with demographic questions and an assessment of drinking behaviour. The main control group undertook a baseline alcohol assessment, and the delayed control group only provided demographic details at baseline. The intervention and main control group were followed up at 6- and 12-months, and the delayed control group at 12-months only.

Eligible participants were undergraduate university students (first and second year students; academic year 2008/9) from U.K. universities, recruited through university information systems (posters, email messages, bulletin boards) and through online social networking sites. Participants were incentivised to respond through entry into a prize draw, with prizes including games consoles, mp3 players and store vouchers.

Ethics approval for this study was provided by Oxford Brookes University Research Ethics Committee (UREC No. 2006/28). Participants gave their informed consent to participate by completing the web-based questionnaire.

The brief personalised normative feedback (PNF) emailed to each participant in the intervention group one to two weeks post-baseline comprised the results of their drinking behaviour assessment compared with, in an easy to read graphical format, average levels of drinking amongst their student peer group. The feedback also provided general information about alcohol and how it might affect them at their current drinking levels, including how long it could take to return to a zero blood alcohol level after a typical drinking occasion. Information was also provided on the money that they might be spending annually on alcohol and also the calories they might be consuming at their current drinking levels. Details of recommended sensible drinking levels were also provided. The composition of the PNF was informed by online normative feedback provided by the Centre for Addiction and Mental Health (CAMH), Toronto [28], and the Brief Alcohol Screening and Intervention for College Students (BASICS) [29] programme. A specimen feedback is available to download [30].

The main objective of the trial was to assess the effectiveness of PNF with risky drinking college students for the prevention of alcohol misuse. A second objective was to assess the relative effectiveness of whole population (universal) versus screening and brief intervention (SBI; targeted) normative feedback in reducing alcohol related problems.

A copy of the online questionnaire is available for download [30]. Baseline demographic questions asked about university year, age, gender, living arrangements and weight.

Respondents completed the Alcohol Use Disorders Identification Test (AUDIT) which is a 10-item scale with good validity that is designed to assess hazardous and harmful drinking [31]. The AUDIT score was specified as the primary outcome variable. In another study we have shown that a small change in AUDIT score may have an important impact on population levels of alcohol disorders [32]. The AUDIT scale is scored between 0 and 40, with a higher score indicating higher levels of drinking.

Frequency of alcohol consumption was assessed with one question asking how often the respondent drank, with responses ‘Never’, ‘1–2 per year’, ‘Once a Month’, ‘Twice a Month’, ‘Once a Week’, ‘Twice a Week’, or ‘Daily’. Responses were calculated as number of drinking days per month; for example ‘Never’ scored 0, ‘Once a Week’ scored 4.5, and ‘Daily’ scored 30.

Usual quantity of alcohol consumption was assessed with one question asking how many drinks/units a respondent usually consumed on a drinking occasion, with responses 0, 1, 2, 3, 4, 5, 6, or 7 or more.

Respondents also completed a drinking diary about a “typical week”, where they indicated the number of drinks/units they usually drink on each day of the week, and this was used to calculate the number of units consumed each week. Respondents were then categorised according to whether they were drinking more than 14 units a week for women and more than 21 units a week for men [33]. In the analysis, respondents were scored 1 if they were over these levels, otherwise 0.

Young people who engage in one problem behaviour (e.g. alcohol misuse) are also more likely to engage in other problem behaviours [34]. Therefore, other problems were measured in a newly-developed self-reported scale with nine possible problems, listed on a yes/no scale: 1. Blackout or memory lapse; 2. Been embarrassed by your actions; 3. Been in a fight; 4. Engaged in unprotected sex; 5. Missed a lecture/class; 6. Required emergency medical treatment; 7. Sustained an injury; 8. Trouble with local or campus authorities; 9. Received unwanted sexual advances. Responses were summed to provide an alcohol-related problems score (range 10 to 20 with a higher score indicating more problems). The internal consistency for this scale was alpha = 0.71.

Perceived norms were measured using an adaptation of the two versions of the Drinking Norms Rating Form [35]. We calculated normative misperception from two variables: the number of alcoholic drinks (“yourself” and “students in your year”) respondents felt were on average consumed at parties or social events. The difference between “yourself” and “students in your year” ranged between −7 and +5, with a negative score indicating that respondents thought that other students were drinking more.

Positive alcohol expectancies were measured using an abbreviated form of the Alcohol Expectancies Questionnaire for Adolescents (AEQ-A) [36]. This self-report questionnaire assesses positive expectancies related to alcohol consumption (social changes, cognitive improvement, sexual enhancement and relaxation) with 21 statements and a true/false answer format. Scores ranged between 0 and 21, with a higher score indicating more positive expectancies.

Several measures were included in the questionnaire to be used as covariates in analyses, including demographic factors (age, gender) as well as baseline consumption. Social desirability responsiveness was also assessed using the short form of the Marlowe Crown scale 2 [37].

We needed an achieved sample size of 900 hazardous drinkers (150 per gender per group), based on effect size estimates of a mean difference in AUDIT score of 1.9 (s.d. = 6) for females [38], and power = .9 and α = .05 (2-tailed tests). Taking account of known prevalence rates for hazardous drinkers in this population, and with cautious estimates for participation and attrition rates, we aimed to recruit 4000 students.

Participants who provided informed consent were individually randomised to intervention or control groups via concealed centrally-allocated computer generated random numbers, with a 1∶1∶1 allocation ratio, without stratification or blocking on any variables.

The computer-based randomisation ensured that researchers and participants were not aware of allocated group.

Participants were recruited via a web site that provided trial information and enabled informed consent. Once consent was given, participants were randomized by computer. There was no direct human involvement in the randomization process.

At randomisation participants were not aware of the nature of the intervention, but full participant blinding was not possible. Similarly, the postgraduate researcher who emailed the standardised PNF to the intervention group participants was not blind to group allocation. All questionnaires were completed remotely via web-based forms, and so outcome assessment was blinded.

Data from the remotely completed online questionnaires were automatically entered and stored in a web based server. At the end of the trial a computer science specialist de-encrypted the data and all personal identifying details were removed from the dataset.

All analyses were intention-to-treat (ITT) and all participants were followed up regardless of their compliance with the intervention. Analyses were performed on the full sample (all students) and also with a higher risk sub-sample to test the prevention paradox prediction. Higher risk drinkers were those who, at baseline, scored 8 or more on the AUDIT scale (the cut-off for hazardous drinking) and also drank more than the recommended weekly consumption limits.

The collected scores about the “Frequency of drinking” and the “Quantity of drinking” were analysed with a mixing distribution of the Poisson regression with a Gamma mixture for panel data using the Stata 10.1 (StataCorp, College Station, Texas). For the proportions of students exceeding 21 units a week (men) or 14 units a week (women), we used generalized linear mixed models with the xtlogit procedure. All other scores (Audit, Alcohol-related Problems, Drinking Norms and Positive Expectancies) were analyzed with linear/log-linear regression using xtmixed (StataCorp). All models included a random intercept to account for clustering within participant (same university) as well as fixed effects for group, follow-up assessment and their interaction. The interactions were removed from the analysis due to insignificant effects. The results are presented as relative risks, odds ratios and difference in regression coefficients, respectively.

We fitted two different random effect models to assess an intention-to-treat hypothesis. In the first random effects model we treated all missing values as missing at random (MAR). Random effects models [29], [30] allow the incorporation of all participants with at least 1 follow-up observation. This model will yield unbiased estimates of the treatment effect under the assumption that values are missing at random. In the second random effects model we carried out a sensitivity analysis using multiple imputation. A chained equation imputation model was fittedsimultaneously for all outcomes as well as baseline AUDIT score, age, and sex, to create imputed complete data sets. Values were imputed for all participants, even for those with no post-baseline data. The following assumptions [39] guided the multiple imputation: 1. all randomised participants were followed-up even if they did not participate in the intervention (i.e. they withdrew from the allocated treatment); 2. We performed a main analysis of all observed data that are valid under a plausible assumption about the missing data; and 3. We performed sensitivity analyses to explore the effect of departures from the assumption made in the main analysis.

The sample at baseline (N = 2611) consisted of first and second year students enrolled in UK universities. Recruitment, follow-up and attrition are described in Figure 1. Follow-up rates were low, with only 50% responding at 6-months and only 40% responding at 12-months. We were not able to collect information about reasons for non-completion of follow-up questionnaires.

Students were recruited into the study at the beginning of the academic year 2007/8, and followed up for 6- and 12-months.

Sample characteristics are described in Table 1. Table 2 describes participants lost to follow-up, with very similar rates in both intervention and the main control groups, suggesting that there was no differential or systematic bias in attrition.

Intention to treat was applied in all analyses. In the completed case analyses (i.e. follow-up respondents only) for high risk drinkers 104/116 universities and 550/1187 individuals were followed up and analysed at 6-months, and 97/116 universities and 444/1187 individuals at 12-months. In the full population analyses 111/122 universities and 876/1751 individuals were followed up and analysed at 6-months, and 107/122 universities and 718/1751 individuals at 12-months.

We have not included the delayed control group in attrition assessment or statistical analysis of effects because there were no statistical differences between the main control and the delayed control groups at 12-month follow-up for any drinking behaviour measures (frequency of drinking, χ² = 6.29, df = 6, p = 0.39; usual quantity of alcohol, t = 0.075, df = 699, p = 0.94; AUDIT, t = 0.63, df = 699, p = 0.53; alcohol-related problems, t = −0.181, df = 699, p = 0.86; perceived drinking norms, t = −0.609, df = 699, p = 0.54). This analysis indicates that there was no effect on drinking behaviour of the baseline measures and questions about alcohol use and problems.

Six- and twelve-month follow-up analysis in the high risk sub-sample showed no effects of the intervention, at either time-point (Tables 3 and 4) and in both the completed case analysis and the multiple imputation analysis, for most outcomes. Only one outcome measure, weekly drinking, showed a significant effect at 6-months with an odds ratio (OR) of 0.417 (95% CI 0.223, 0.781), but this had disappeared by 12-months (OR = 0.710, 95% CI 0.435, 1.160). The 12-month significant effect remained in the multiple imputation analysis, but this should be regarded with caution given the very high attrition rates. Moreover, this one effect should not be over-interpreted given the consistent pattern of no effect across most outcome measures and time points.

In the full sample analysis there was a similar pattern of no effect of the intervention (Tables 5 and 6) at either time-point and in both the completed case analysis and the multiple imputation analysis. Again, only weekly drinking had a significant effect at six-months (OR = 0.440, 95% CI 0.245, 0.788), but not at 12-months (OR = 0.770, 95% CI 0.495, 1.197). Overall, there is a consistent pattern of no effect across most outcome measures and time points.

No ancillary analyses were undertaken.

No adverse events were reported.

The innovative design and sampling aspects of this study: the Solomon three-group design to test for measurement effects, and the whole sample and higher risk sub-sample analyses to test the prevention paradox prediction, were illuminating. We found no evidence that simply asking people about their drinking behaviour, in the absence of any intervention, led to any change in behaviour. Similarly, we found no evidence that a population wide intervention was more effective than a targeted intervention with a higher risk sub-sample of students but, given that we found no effects of the PNF intervention across a range of alcohol outcomes, this is not surprising.

The reason(s) for the inconsistent findings between our study and some other studies is not clear. One line of thought is that the accumulating evidence on social normative feedback is an example of initially positive research findings ultimately being found to be false [47]. Another possibility is that the UK offers a more challenging context for effectiveness because of local contextual factors that are not present in other countries, for example it may be that the prevalence of excessive drinking behaviours in the UK, and the cultural acceptance of heavier drinking, provides a more hostile environment for success of normative feedback interventions. Alternatively, the intervention we used, based on studies from other countries, may not have been sufficiently developed for the UK population. In this study personalised normative feedback was presented to university students via email with information about the norms for the “average” student in their university, as in previous research [48] where similar proximal referent norms were used. Research has demonstrated that when compared with more distal referents, proximal referents are more effective for preventing student alcohol misuse and related problems [35], [49], [50]. On the other hand, a study in Canada [51] found that the social norm materials that were designed to be the least comparable had the greatest impact, and that Canadian students were more familiar with American drinking norms because of media influence, with television being the most significant medium for the spread of youth culture.

We should also point out that, like many other studies that have reported the effects of brief personalised normative feedback, the intervention contained more than just the normative feedback. We also provided financial feedback, health information and advice on where help can be obtained. The interaction between these different components, and how they are presented to participants, may be important in determining effectiveness. Clues as to the differences in effect across different studies might lie in the differences in intervention content and delivery.

Attrition in this study was lower than in other European research on email- or web-based social normative feedback interventions with university students [44]–[46] though higher than in some Australian or U.S. studies [40], [42]. There is a potential risk to internal validity from the low follow-up rates achieved in our study, although students unavailable for follow-up were similar across groups with regard to sex, age and baseline drinking status, and the multiple imputation sensitivity analysis did not produce any marked or systematic changes in treatment effect sizes or significance. But, due to the low follow-up rates, we cannot absolutely rule out the possibility that differential attrition in relation to unmeasured and uncontrolled confounders could have affected the results.

One possible reason for the low retention rates in our study is that the personalised feedback given in our study was delayed by a few weeks, as we needed to collect and analyse all baseline data to provide a normative comparison dataset for personalised feedback to each participant. It is conceivable that this could have influenced retention rates, but we regard this as unlikely given more immediate feedback was provided in similar studies that had similar or poorer retention rates [44]–[46] and similar findings of limited/no effectiveness.

A more serious issue is in relation to external validity, specifically generalizability [22]. It is hard to see how the sub-set of students that were retained in the study are generally representative of the whole student population, so inferences from sample to the pre-specified study population are problematic. In fact, the low-follow-up rates add to the problem with understanding how generalizable these study results are given the recruitment methods used: only those students who saw the study adverts (via email, student information systems or Facebook) were able to participate and, as the most effective recruitment strategy was Facebook with 78% of all participants, it is unclear how representative these participants are of the general student body. So, one possible explanation for our null results in this study is that we obtained results on a different group of students than other studies that have found significant effects.

On the other hand, this study was a large pragmatic randomised trial with design, sampling, recruitment and follow-up characteristics that are similar to other large European trials [44]–[46]. All these other European trials have had low follow-up rates from those randomised and assessed at baseline, and with similar non-significant effects to our study. An important conclusion to draw out from our study, alongside these other European studies, is that the acceptability and viability of recruiting students into a brief personalised feedback intervention, outside of a university medical centre screening programme [40] or a mandated student alcohol education programme [42], seems to be low.

In conclusion, our results show no evidence for the effectiveness of personalised normative feedback for the prevention of alcohol misuse and alcohol-related problems in a UK student population. The applicability of personalised normative feedback to populations and settings that are different from those in the trials whether positive effects have been found is uncertain [22]; moreover an overall conclusion of no effect for this sort of intervention [47] cannot be discounted.