The Mayo Clinic institutional review board deemed this study exempt from full review. We recruited participants via email. We obtained verbal consent at the start of each session as approved by the review board, and documented this in the session minutes (written consent is not required by our review board for studies of this type).

From October 2011 to February 2012 we conducted 11 focus group sessions at an academic medical center and four affiliated primary care sites (see Table 1). At the start of each session the moderator read a brief clinical scenario describing a hypothetical patient with “several medical issues.” The moderator then noted, “You realize you’re uncertain about how to proceed with managing a specific issue,” and asked participants, using an interview guide,[5] to explain how they resolve such uncertainties.

Focus group participants included 50 practicing physicians in a large multi-site health system (see Table 1), selected through purposive sampling to reflect diverse backgrounds (internal medicine and family medicine physicians; generalists and subspecialists; and academic and community practitioners). We determined sample size using thematic saturation: after the first six focus groups, and after every one or two thereafter, we reviewed moderator notes and transcripts to identify newly-emergent themes. We stopped scheduling sessions when no new themes emerged.

All providers in our health system enjoy free access to a variety of commercial online knowledge resources including UpToDate (a physician-authored resource), MD Consult (a compilation of full-text journal articles, medical references, and drug information), and Micromedex (a collection of databases focused on drugs and toxicology). They also have access to a locally-developed evidence-based resource, AskMayoExpert, that contains care process models (algorithms describing ideal care pathways), answers to frequently-asked clinical questions, and contact information for topic experts.

After each session participants completed a brief questionnaire about their use of online resources. Each session was audio recorded and transcribed verbatim except for masking of participant names. We analyzed the transcripts using the constant comparative method.[29] Investigators DAC, KJS, and RAB first inductively identified main themes in a high-level review of raw transcripts. Investigators DAC and KJS then coded all transcripts, iteratively refining (merging and splitting) the initial themes to accommodate emergent concepts, and continuously contrasting themes and excerpts to identify interrelationships among themes. We then consolidated these themes into a coherent model that parsimoniously reflected these interpretations. We used Dedoose (www.dedoose.com) to facilitate the coding and analysis.

In a brief survey at the end of the focus group, nearly all of the participants (46/50) reported using one or more online references at least twice weekly. These included UpToDate (N = 37), PubMed or MEDLINE (N = 18), specific online journals (N = 10), and AskMayoExpert (N = 10). Half (N = 25) reported using a mobile device (smartphone, tablet computer, etc) on a daily basis for work-related activities.

Participants noted strengths and weaknesses for several specific knowledge resources, as summarized in Table 2 (see Table S1 for supporting quotations). Notably, the online resource UpToDate was by far the most commonly cited in both our survey (above) and in focus group discussions. While physicians lauded UpToDate's comprehensiveness, grounding in evidence, and relatively efficient navigation and search functionality, they generally agreed that UpToDate was far from perfect. Most notably, they perceived this resource generally presents too much information, and the information occasionally fails to answer common clinical questions.

The locally-developed AskMayoExpert resource (see description in Methods) was discussed by many physicians as the chief competitor to UpToDate for answering clinical questions at the POC. Concise, practical answers and relevance to local practices were noted as the chief strengths, but spotty topical coverage and lack of explicit reference to supporting evidence/literature were cited as weaknesses. Physicians also noted difficulty navigating or searching the site. Several physicians disliked the question-and-answer format, although others noted this as a strength.

Google was often cited as the next source. Physicians appreciated its speed, availability, intuitive search, and access to images and other resources. However, they expressed concern about the variable credibility of the sources. MD Consult was also frequently noted, but typically as a third- or fourth-line resource because of its difficult navigation and search functionality and because of the “keyhole effect,” i.e., presenting a narrow slice of information such that piecing together a complete answer is difficult. Online literature indices such as PubMed and MEDLINE were frequently mentioned, but because of the time required to search for and digest information they were rarely described as useful in POC learning, unless the physician knew a priori of a specific article (e.g., a clinical guideline). However, for learning after the patient left or for general learning needs, physicians noted that literature indices possess numerous strengths including credibility, currency, and capacity for obscure questions. Several physicians noted Google Scholar as a quick way to find credible information, including full-text journal articles and guidelines.

Despite the growing presence of electronic resources, several physicians noted continued regular use of print materials such as textbooks and article reprints. One of the most consistent strengths of these resources is physicians' familiarity with these texts, which in turn facilitates rapid identification of needed information and (for information they had previously studied) learning reinforcement. Participants also described the benefit of stumbling across information on a nearby page that informs their decisions, but that they might miss in a targeted (i.e., electronic) search (the opposite of the keyhole effect). Challenges in using print materials as point-of-care resources include their physical inaccessibility (generally felt to be a minor concern) and the volume of information in most textbooks.

Finally, many physicians use personally-developed resources on a regular basis, both print (e.g. index cards) and electronic (on a mobile device, e-mail folder, or computer server). Advantages of personal resources include high familiarity and speed of access, relevance to needs, and a personalized organization structure.

Drawing on the strengths and weaknesses noted above, together with other comments, we identified nine key features of useful knowledge resources (see Table 3).

These features do not define an ideal knowledge resource, but do propose criteria for more effective resources. Our study design did not permit objective evaluation of the relative merits of each feature; this could be the subject of future research. While the specific comments referring to a proprietary system (AskMayoExpert) are of limited generalizability, they provided a useful contrast with other resources and thus informed our model; we have already made substantial improvements to this system in response to these comments. Physician opinions might not necessarily indicate effective practices, and we did not triangulate our focus group data with other observations. Strengths of this study include the rigorous application of qualitative techniques, and the purposeful involvement of primary care and specialist physicians in both academic and community settings.

Studies have evaluated knowledge synthesis resources in comparison with other knowledge resources,[30]–[33] PubMed,[11], [12], [33] and no specific intervention.[13], [15], [16] Others have evaluated the topical coverage, evidence base, and currency of various resources.[18], [19], [34]–[37] However, none of these studies systematically evaluated or identified important design characteristics.

Ely[7] identified 22 recommendations to improve knowledge resources, clustered in the categories of comprehensiveness, trust, navigation, clinical organization, and accessibility. Our work complements this prior work by re-emphasizing several common recommendations and by adding new key features: integrated, familiar, capable of identifying the human expert, and reflective of local care processes.

Given the prevalent use of Internet resources like Google,[20], [21] it seems appropriate to study how to judge the quality of the information thus obtained[38] and how to train physicians to make such judgments.[39]

Clinicians have difficulty answering complex questions using electronic knowledge resources.[27], [37] While one solution is to simplify the question,[27] participants in our study indicated that for complex questions they typically contact a human expert. Expanding the definition of knowledge resources to include curbside consultations[22] enables new opportunities to facilitate knowledge transfer and POC learning.

Previous studies have identified several barriers to physician information seeking other than the availability and functionality of knowledge resources, including insufficient time, patient complexity, inadequate search skills, the sheer volume of information available, and belief that an answer is not available.[2], [5]–[10] Providing access to robust knowledge resources without addressing these other concerns may be insufficient to improve POC learning.

Given the growing volume of medical information,[40] physicians will of necessity increasingly turn to evidence syntheses[41] to guide practice change. Although the rigorous grading and distillation of such evidence is essential, the manner in which this information is then communicated to physicians is likewise important.[27] The features identified in this study offer guidance regarding how this might be effectively done. While our model does not rise to the level of theory, it nonetheless incorporates and integrates several disparate perspectives of informatics and clinical decision-making including decision support, human factors, evidence-based practice, local care variation, point-of-care learning, and inter-physician communication. Concurrent with design improvements, developers might evaluate the comparative effectiveness of the proposed features to confirm their merit and determine their relative costs and benefits.