Uncovering impacts: a case study in using altmetrics tools

Altmetrics were born from a desire to see and measure research impact differently. Complementing traditional citation analysis, altmetrics are intended to reflect more broad views of research impact by taking into account the use of digital scholarly communication tools. Aggregating online attention paid to individual scholarly articles and data sets is the approach taken by Altmetric LLP, an altmetrics tool provider. Potential uses for article-level metrics collected by Altmetric include: 1) the assessment of an article's impact within a particular community, 2) the assessment of the overall impact of a body of scholarly work, and 3) the characterization of entire author and reader communities that engage with particular articles online. Although attention metrics are still being refined, qualitative altmetrics data are beginning to illustrate the rich new world of scholarly communication, and are emerging as ways to highlight the immediate societal impacts of research.


Introduction
The future of scholarly communication is one in which a large part of scholarly communication is conducted online [3].A key part of the scholarly communication lifecycle is trying to understand the impact of work.The process of understanding impact helps scientists, science administrators and others both find, evaluate, and access scholarly products.Traditionally, this impact assessment has been done primarily through the tracking of formal citations.This is possible because citations counts, for all their occasional ambiguity [2], do reflect use of scholarly products.However, this reflection is of a restricted spectrum; scholarly products are often used by scholars, and others, in ways that do not perturb the citation record [5].Furthermore, traditional citation does not reflect the rapid nature of communications afforded by the Web.Thus, we need new approaches for measuring impact in this changed world.
Indeed, because of the Web scholarly communication, formerly "underground" uses like reading, bookmarking, sharing, discussing, and rating are beginning to leave online traces.The are becoming visible on Web pages [8,13], on blogs [6], in downloads [1,4], on social media like Twitter [9], and in social reference managers like CiteULike, Mendeley, and Zotero [7].These alternatives to traditional citation analysis have been labeled altmetrics [11].Altmetrics offer potential for gathering information on more diverse types of impact, from more diverse scholarly products, including blog posts, slides, datasets, or even tweets.They also have the important benefit of speed; altmetrics typically accumulate in days or weeks rather than the years citations require.This is particular useful in as the research process increases pace where users of scientific content need to understand the impact of it rapidly.To begin to make practical use of altmetrics for measuring impact requires both a greater understanding of the properties and validity of these new metrics, and practical tools for obtaining them [10].Others have begun the former [12]; here we will pursue the latter, presenting two new tools for gathering and presenting altmetrics.
2 Tools for Altmetrics: CitedIn and total-impact CitedIn (http://citedin.org)and total-impact (http://total-impact.org) are opensource tools that receive as input a list of identifiers for scholarly products, and output a set of altmetrics for each product.CitedIn accepts only articles with PubMed IDs (PMIDs); total-impact accepts articles identified by PMID or DOI, but also datasets and slides using a variety of identifiers including URL, handle, and accession numbers.Both tools allow users to input identifiers manually; CitedIn also offers a REST API, and total-impact lets users automatically populate the products list using items stored in Mendeley or Slideshare libraries.Once users have uploaded products, CitedIn and total-impact both use calls to open Web APIs to gather data about them; CitedIn also caches available databases.As of September 25, 2011, the data sources used by each are listed in Table 1.
In addition to gathering altmetrics from these sources, both tools also include some additional features.CitedIn lets users input and output data over a REST API, and also reports a "CI-number" that summarizes all almetrics activity in a single value.Total-impact offers persistent URLs for impact report pages; the impact metrics can be refreshed over time.Both tools let users download results as structured text files for further analysis.Output pages for the tools are shown in Figures 1 and 2.

Case study: altmetrics for a national research center
We used a set of 214 articles from the National Evolutionary Synthesis Center (NESCent) as a realistic test for the two tools.NESCent was interested in tracking the impact of work they funded in a faster and more comprehensive way than citation analysis allowed -a typical use case for altmetrics.We entered the articles into CitedIn on August 14 2011, and into total-impact September 23 2011, then collected and analyzed the results.
All 214 articles had DOIs, and so were able to be processed by total-impact.Only 174 articles had the PMIDs required by CitedIn, so the CitedIn sample  is smaller.Both tools showed that altmetric activity as measured by number of "altmetric events" (bookmarks, downloads, etc.) is relatively widespread across articles: CitedIn found at least one event on 95% of its articles, and total-impact on 85%.There were a mean of 28 and median of 16 events per CitedIn article, with a maximum of 678.Total-impact had a per-article mean of 92 events and a median of 19; the higher mean is due to Dryad dataset downloads, which accumulate more easily than other metrics, reaching a maximum of 2769 on one article.We visualized the activity across articles using heatmaps, shown in Figures 3 and 4 to create a sort of "impact genome."Only altmetrics with nonzero counts are shown, and counts of each altmetric are normalized by that metric's maximum.Articles are arranged so that those with higher mean event counts across all metrics are further left.

Conclusion
Altmetrics have potential to improve the speed and breadth of scientific evaluation.CitedIn and total-impact are two tools in early development that aim to gather altmetrics.A test of these tools using a real-life dataset shows that they work, and that there is a meaningful amount of altmetrics data available for use.These tools continue to improve: check out the current versions for up to date capabilities.
The properties and validity of these data, however, are still unclear, and call for additional research.What is the scholarly value of, for instance, a Mendeley bookmark or a Wikipedia citation?Future work should also investigate how altmetrics for different sets of articles can be compared; this is a particularly tricky problem given the high dimensionality of altmetrics data, and may benefit from better visualization techniques, or statistical approaches like principle component analysis and factor analysis.

Fig. 4 .
Fig. 4. Active total-impact event types and normalized event counts per article.

Table 1 .
Data sources for CitedIn and total-impact as of September 2011