Based on our own monitoring of scientific articles and news media, as well as consultations with influenza experts, primarily from WHO, we identified five terms as relevant and likely to be present in the records of articles on the pandemic: “h1n1,” “swine,” “pandemic,” “epidemic,” and “outbreak.” We considered the last four relevant only if the term “influenza” or “flu” was also present in the database's record of the article. The following steps accordingly made up our core PubMed search string:

Because of PubMed's automatic term-mapping feature, certain terms are “translated” when searches are carried out [6]. For the above search, the following translations apply. (See Box S1 for an explanation of the search tags we used.)

This translation ensures that records indexed under the included MeSH “Disease Outbreaks,” “Influenza, Human” or “Swine” are identified even if they do not contain any of the actual terms used in the search algorithm. It is worth noting that the MeSH “Influenza, Human,” which both “influenza” and “flu” translate into, does not include the MeSH “Influenza A Virus, H1N1 Subtype” as these MeSH are placed in two separate branches of the vocabulary's hierarchical structure. However, including the term “h1n1”[all] or “influenza”[all] will return records assigned the latter MeSH because it contains both “h1n1” and “influenza.”

Using the core search string, we built additional strings to simulate a search carried out at the end of a given week for relevant materials that had been added to PubMed during that week. We used weeks that ran from Monday to Sunday, beginning with the last week of April 2009, meaning that Week 1 was 27 April–3 May 2009, Week 2 was 4–10 May 2009, and so on.

To find all the records available at the end of Week 1, we first used PubMed to identify all records returned by the core search string:

We then limited the records to the ones entered into PubMed during Week 1:

For the next part of the algorithm, we looked for records with a MeSH date (see Box S1) between 27 April and 3 May 2009, which identified all records that had been assigned MeSH during Week 1 or had been entered into PubMed in Week 1 but not yet assigned MeSH at the time of our simulated search.

Then we isolated the records that have been assigned MeSH after Week 1, i.e. later than 3 May 2009:

To these we applied the core search string as if no MeSH had been assigned to the records, to reflect the delay in assigning such headings. In a typical search conducted at the time, almost all the database fields would have been searched, including MeSH (see the description of the search tag [all] in Box S1). Ideally, we would restrict the search to all database fields besides MeSH, but PubMed does not provide a possibility to exclude a search tag, such as [mh]. Instead, we limited our search to the fields “Title” and “Abstract” (by using the search tag [tiab]), assuming that no other fields would contain any information that would retrieve a record if a title and abstract search did not already identify it.

As the automatic translation produces a string that includes additional terms, we included them explicitly in our string. However, we removed terms that would produce results identical to other terms in the string, such as “sus scrofa,” “disease outbreaks,” and “human influenza.” The resulting search simulated one in which MeSH have not yet been assigned to the records.

Finally, we joined the results of our search for records that have not yet been assigned MeSH or where this happened during Week 1 with the records assigned MeSH after Week 1:

The end result simulates a search carried out at the end of the last day of Week 1, i.e. 3 May 2009, for potentially relevant publications added to the PubMed database during the previous week.

By reviewing the titles and available abstracts of all the potentially relevant records returned by Step 10, we identified the ones relevant to the 2009 influenza A(H1N1) pandemic. We conducted this review individually, and then the two of us conferred on which records to classify as relevant.

We included as relevant articles accepted for publication or published before the first outbreak was known if they were on subjects such as influenza vaccine development, oseltamivir (Tamiflu) resistance, or general influenza pandemic preparedness. We excluded articles primarily on influenza A(H5N1) (avian influenza) if they contained no obvious linkages to one of these subjects.

We considered all types of publications for classification as relevant, including publications categorised by the NLM as “news,” which covers announcements, statements of new data, reports of recent events, and other matters of interest to the field of science. Nature and Science are two examples of journals publishing substantive news reports of vitally important and sometimes controversial developments, often with data [7].

Then we incorporated the ID of every relevant record in a single search string. This query produced a list of the relevant records that would have been found if the core search string (#1) was used in a PubMed search at the end of a given week, e.g. at the close of 3 May 2009, for articles added to PubMed during that week.

To show how many relevant records would not have been identified if one of the key terms – “h1n1,” “swine,” “influenza,” or “flu” – were left out of the search string, we carried out a search that was similar to the above but excluded each term in turn, and then we compared the results. The following example excludes “h1n1”:

We carried out similar searches that excluded the other key terms in turn. For “swine,” the term itself was excluded as well as traces produced by the automatic translation, (“swine”[tiab] OR (“sus”[tiab] AND “scrofa”[tiab])). For “influenza” and for “flu,” we only excluded the term itself, since in both cases the translation adds (“influenza”[tiab] AND “human”[tiab]) to the search string.

As a tool for additional analysis of parts of the results we extracted the publication types assigned to the records. We divided the publication types relevant for this study into two categories:

Records can be assigned one or more publication types. This is not necessarily done at the same time as MeSH are added.

The principal limitation of this study is the difficulty of validating the results. It can be done in part by “hand-searching” all available journals during the 10-week study period. Yet that would involve individually searching not only the journals that returned results for our core search string, but all the journals in PubMed. As such it is clearly not an option. Moreover, since MEDLINE indexing can take several months, many journal issues from the study period are still not indexed in the database and would thus require physical inspection.

There are several ways to improve the success of PubMed searches for pandemic information. First, the NLM could provide journal publishers with the possibility of including a special alert tag when they upload a new citation to the PubMed database. This tag would indicate that the cited item includes information about a rapidly evolving situation such as a pandemic. If these tags were used consistently, they would greatly facilitate such searches. The tags could then be revised or removed when MeSH are added. To our knowledge, this approach has not previously been tested in the context of health related bibliographic databases. However, a similar concept is widely used for information sharing on Internet services like CiteULike (http://www.citeulike.org) and Delicious (http://www.del.icio.us), where users can add tags of their own choice to their papers and links.

Second, publishers should be more diligent about including short abstracts of the records they add to PubMed. As shown, a substantial portion of the relevant records that were missed did not have an abstract. An abstract increases the likelihood that a searcher will find – and utilise – relevant materials.

Third, as part of its pandemic preparedness planning, WHO should prioritize the prompt naming of new disease strains involved in outbreaks, after appropriate consultation with scientific experts, research librarians, communication experts, and perhaps linguists. Such action should not hinder the introduction of a new name at a later stage, as long as consistency in use is maintained. A recent study has shown that WHO was the most cited institution during the first days of the influenza A(H1N1) pandemic [14]. As such it plays a key role in determining and clarifying pandemic nomenclature. It is no small responsibility, given that the inability of (re)searchers to find relevant articles impedes the transfer of knowledge to experts and health policy-makers, to the potential detriment of public health.

Fourth, our own experience in creating and maintaining the influenza A(H1N1) web site for the WHO Regional Office for Europe (http://www.euro.who.int/influenza/ah1n1) underscores the importance of carefully and consistently translating key pandemic terms into languages other than English. Although not relevant for the PubMed database, this issue is important for searching the WHO site and other web sites in foreign languages.

Finally, to avoid delays in becoming acquainted with new research, we suggest that people who rely on access to the latest published research subscribe to the really simple syndication (RSS) feeds provided by most publishers and PubMed itself. This technology can be utilised in several ways to enhance the timeliness and retrieval of research updates (see Box S2 for an example).

Researchers and other experts should realize that when they search for newly published scientific material at the beginning of a pandemic, such as the 2009 influenza A(H1N1) pandemic, it is highly probable that they will not retrieve all the relevant articles. Our study demonstrated that a search string that does not include a combination of terms that covers several of the names used for the pandemic will miss relevant publications indexed in PubMed/MEDLINE. Leaving out just one term from a search can result in missing as much as a third of the relevant articles at a given point in time. These findings can have significant implications for the communication and utilisation of pandemic information.

There are two main ways to remedy this deficiency. The first is to agree on a name earlier and clearly communicate it (and any subsequent changes) to the scientific community and media. The World Health Organization is perhaps best placed to lead this effort. The second solution is for the National Library of Medicine to implement faster indexing of publications that relate to a rapidly unfolding health crisis, as well as to provide publishers the possibility of adding alert tags for such articles.