For patients to be able to optimally manage their health, they require an adequate level of understanding about their condition and associated management strategies [1–3]. This is related to an individual’s level of health literacy and is an increasingly important area of research particularly in people with chronic disease. Cardiovascular disease (CVD), for example, is a major health burden for patients and often requires life-long behaviour changes across multiple risk factors [4]. CVD management depends on active patient participation, which requires a sufficient level of health literacy [5]. It has been shown that patients with CVD that have a low level of health literacy are less likely to adhere to prescribed medications [4–6]. Similarly, lack of adherence to prescribed medication has also been found to be associated with inadequate or marginal health literacy [4].

Electronic health (eHealth) literacy has become increasingly relevant in recent years with the development of eHealth tools to support healthcare delivery and management [7–9]. However, with these innovative developments eHealth literacy becomes an equally important area of investigation. eHealth literacy is defined as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem” [10]. The additional challenge of eHealth relates to the skills required to use an electronic device as well as having an adequate level of health literacy to effectively make decisions regarding health [2,9]. In order to ensure the existing scales for health and eHealth literacy are clinically applicable, it therefore becomes important to examine how they perform in various populations when developing, evaluating and implementing health management strategies.

Several scales can be used to measure health literacy such as the Test of Functional Health Literacy in Adults (TOFHLA) [11] and the Rapid Estimate of Adult Literacy in Medicine (REALM) [12]. However, these scales assess different areas of health literacy which means that an individual can have a very different level of health literacy depending on the scale used [13]. A more comprehensive measure of health literacy is the Health Literacy Questionnaire (HLQ) [14]. In contrast, there are currently very few scales available to evaluate eHealth literacy [7]. One such scale is the electronic Health Literacy Scale (eHEALS) [15]. The psychometric properties of both the eHEALS and the HLQ have been examined using classical test theory and item response theory methods. Prior studies using confirmatory factor analysis (CFA) have demonstrated the eHEALS to be a valid and reliable scale [16,17]. More recent item response theory studies have confirmed these findings [18,19] and demonstrated that the scale measures the same underlying concept or construct. However, a recent study found that the eHEALS was not able to capture the full range of eHealth literacy levels in their study population (i.e. ceiling and floor effects) [18]. The factor structure of the HLQ has also been tested with CFA and three subscales tested were found to be measuring the same underlying construct [19]. It has also demonstrated satisfactory reliability [14]. However discrimination between response categories was questioned for three of the nine scales [14,20], suggesting that response categories could be revised to improve the scales measurement properties.

There is growing evidence to indicate that the Rasch measurement model, which measures a scale’s internal construct validity, is the gold standard for psychometric evaluations of outcome scales [21]. Rasch analysis has several well-recognised advantages over classical test theory including analysis of item summation legitimacy, response category distinction, hierarchical structure of item difficulty and discrepancies in item response for a given level of ability [22,23], which all strengthen scale construct validation. Having valid scales is particularly relevant in a population at risk for CVD where an adequate level of health literacy has been shown to improve effective patient disease management [24]. Moreover, there has not yet been a Rasch validation study in a population with CVD. Given that CFA may not fully resolve issues associated with the conceptual structures of psychological scales such as health literacy [20], further evaluation using the Rasch measurement model is required to extend the current knowledge on the validity of these scales. The aim of this study was to extend prior validation studies by examining the internal construct validity of an eHealth and a health literacy scale using Rasch analysis to provide clinicians and researchers with information on the usefulness of these scales.

Demographic characteristics of the 397 consenting participants are presented in Table 3. Three-quarters of the sample were male, with a mean age of 66.3 years (SD 8.1), 88% were Caucasian and the majority (79%) were married or in a defacto relationship. Due to incomplete scale completion, one participant was removed from both the eHEALS and HLQ for the Rasch analysis. The mean total score for the eHEALS was 27.1 (range: 8–40; SD 6.67), 3.03 (range: 1–4; SD 0.52) for the first five HLQ subscales and 4.19 (range: 1–5; SD 0.47) for subscales 6 to 9 (Table 4).

This study found that the eHEALS had good overall fit to the Rasch model, distinct response categories, no item bias and reasonbale targeting. This provides evidence supporting the use of the 8-item eHEALS as a measure of eHealth literacy with higher scores truly representing higher levels of eHealth literacy. All subscales of the HLQ except for subscale 5, on the other hand, did not fit the Rasch measurement model and had suboptimal targeting. Nevertheless, all subscales apart from subscale 9 did demonstrate unidimensionality, had no major item bias and moderate to good internal consistency. This indicates that the HLQ suitably measures nine aspects of health literacy. Both scales were, however, unable to reflect mid to high ranges of health literacy. Consequently, when they are used to measure changes in ability over time, an individual would have to acquire a substantial increase in health or eHealth literacy for it to be reflected in their score. This could account for the presence of misfitting items and persons and highlights the potential for further scale testing and refinement.

Prior studies using factor analysis or principal component analysis (PCA) and item response theory (IRT) have found the eHEALS to be internally consistent (Crohnbach α: 0.87 to 0.94) with modest to good stability [7,16,17,34] and have supported the construct validity of the scale. The high internal consistency found in this study (PSI: 0.90) further supports these findings [7,16]. A recent study using IRT in a population with diverse chronic diseases demonstrated good model fit and distinct response categories for the eHEALS [19]. Good model fit was also found in another IRT study (in a student population and adults who use the Internet) with similar findings around targeting, notably a marked ceiling effect [18]. Unlike the current study, however, response categories were not distinct in the ‘strongly disagree’ and ‘disagree’ categories [18]. Further research is needed to test whether reducing the number of eHEALS response categories would better reflect the range of eHealth literacy levels.

Contrary to our findings, previous studies using classical test theory and item response theory (in a student population, healthy adults using the internet and patients with rheumatic disease) have supported unidimensionality of the eHEALS [17,34]. In our analysis, we identified two constructs, or concepts, of eHealth literacy: (1) items 1–5 (around knowledge about resources) and (2) items 6–8 (around evaluation of resources). A Likert-scale, such as the eHEALS, depends on unidimensionality to be able to sum its items [35]. A possible explanation for this discrepancy may be related to the age of the population studied with this study focusing on an older population (mean age: 66 years old). It could be that the knowledge to use the Internet and the ability to evaluate online resources are two distinct skills for an older population but not for a younger one that is more familiar with the electronic medium. Further research is therefore required in different populations to explore whether the eHEALS is indeed composed of two separate constructs, which may then mean summing item scores across constructs is not appropriate.

The HLQ has previously been found to have a good level of construct validity and a composite reliability of ≥0.8 for almost all subscales [14,36]. However, the scale developers had noted that participants had difficulty distinguishing between the response categories for subscales six to nine (‘very difficult’, ‘quite difficult’, ‘quite easy’ and ‘very easy’) [14]. This was confirmed in their follow-up paper using a Bayesian model approach where subscales six to eight were found to have disordered thresholds [20]. Similar findings were observed in this Rasch analyses as subscales six to nine displayed disordered thresholds. Disordered thresholds are not uncommon in scales with multiple response categories or when wording between them is similar [23,35]. If participants are not able to distinguish between response categories, the sum of the items does not truly reflect the individual’s ability i.e. selecting ‘quite’ difficult would be equivalent to selecting ‘very difficult’. The Rasch analysis demonstrated that collapsing two response categories (‘very difficult’ and ‘quite difficult’) resulted in ordered thresholds. This improved the ability for participants to distinguish between the response categories and ultimately, could enable healthcare providers to accurately measure the range of health literacy levels.

This is the first study to have assessed and compared the internal construct validity of the eHEALS and the HLQ scales using Rasch analysis in a population with CVD. Validation of such scales is particularly important in order to accurately measure the health literacy of patients with chronic disease, such as CVD, who are active participants in the management of their health. This study was able to provide new insight into the measurement properties of two commonly used scales for eHealth and health literacy by comparing prior findings. It was also able to provide information regarding item bias (ie. DIF), which has not yet been examined in the eHEALS. Furthermore, Rasch was able to analyse several psychometric aspects of the scale, such as the hierarchical structure of items, ordering of response categories, DIF and the summation of the scale, which are beyond the scope of classical test theory. We were, therefore, able to highlight aspects of both scales which require further investigation such as whether the eHEALS measures two distinct constructs of eHealth literacy and provide further support for rescoring subscales six to nine of the HLQ response categories in order to better reflect hierarchical distribution of health literacy. Further validation is now required to test these changes in a different population.

This research does have several limitations. Firstly, the cohort is mostly male and the group is known to have home Internet access, reducing generalizability of the findings. Secondly, both scales had many misfitting items and persons, which may in turn have contributed to the degree of misfit observed between the data and the Rasch measurement model. However, achieving model fit would have entailed deletion of items or persons, which was beyond the scope of this paper. Thirdly, this study did not seek to confirm whether individuals with higher eHealth or health literacy scores truly had better disease management. This is particularly noteworthy since a Dutch study found that perceived skills as measured by the eHEALS did not predict actual performance [34], although it is the only study to have done this analysis and further research is required to confirm this finding. Furthermore, we were not able to assess the external validity of both scales, which is an important area of health literacy. Finally, Rasch analysis is one among many tools for internal construct validity and the results of this study should be considered among the already existing validation work that has been done on these two scales.

Electronic resources are valuable tools to access health information, however, patients must acquire the skills to effectively engage with and benefit from the plethora of resources available online. This is particularly true for CVD in which disease management and prevention depend largely on patients’ health literacy. For healthcare providers to optimally assess a patient’s level of health literacy, valid and reliable scales are essential. This study demonstrates the good psychometric properties of eHEALS and highlights that the HLQ appropriately measures its nine distinct aspects of health literacy. Further research is now needed to determine the extent to which higher eHEALS scores correctly identify those individuals with greater eHealth capacities and whether collapsing response categories in the last four subscales of the HLQ improves boundaries between response categories.