Understanding Research Impact

Understanding the true value of a scholar’s research and output is no small feat. Although it’s fairly straightforward to track the number of publications or total dollar amount of awarded funding, it can be a greater challenge to assess the reach of scholarly efforts and determine how others are utilizing the research results. Metrics for assessing research performance, quality, and impact cover a wide range of the scholarly ecosystem and are used for a variety of purposes: individual career planning, promotion, and tenure; benchmarking to track group or institutional performance; marketing and strategic planning purposes; and reporting research outcomes to the public.

Author-level metrics

Author-level metrics allow individuals to track their scholarly output and serve as a reflection of a researcher’s productivity. Enumerating such things as the number of publications in the scholarly literature, number of books published, and number and amount of funding awards can all serve to understand the efforts of individual researchers. One commonly used metric is the Hirsch index (h index). The h index, developed by Jorge E. Hirsch, PhD, in 2005 (1), offers a numeric index to measure the productivity and impact of a given researcher. The h index is a quantitative metric based on analysis of publication data, using publications and citations to provide “an estimate of the importance, significance, and broad impact of a scientist’s cumulative research contributions” (1). According to Hirsch, the h index is defined as follows: “A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np – h) papers have ≤h citations each” (1), where Np is total number of papers published.

As an example, an h index of 10 means that among all publications by one author, 10 of these publications have received at least 10 citations each. The h index is but one metric for author-level assessment. No single metric is sufficient for measuring performance, quality, or impact by an author; indeed, the discovery of a scholar’s most impactful work may be gleaned only through qualitative forms of assessment that do not rely solely on publication data.

Article-level metrics

Citation counts are perhaps the most frequently used metric at the article level. A citation is a reference to a specific publication. The inherent assumption is that significant articles will have high numbers of citations. Further analysis is required to discover why select publications garner a higher citation rate than others. Many databases provide tools for authors to track citations to their work, with some offering citation maps that can be downloaded for reporting purposes.

A growing article-level metric is based on the usage of a publication; several journals and third-party service providers are making it possible to assess the Web-based use and subsequent dissemination of individual articles. The Public Library of Science (PLoS) journals offer perhaps the most highly developed publisher-based platform for this type of tracking. Articles published in PLoS journals include an article-level metrics tab that shows such details as article usage statistics (e.g., HTML page views, PDF and XML downloads, and accesses from PubMed Central; number of users via Mendeley; and number of Facebook mentions); citations from the scholarly literature (currently from CrossRef, PubMed Central, SciVerse Scopus, and Web of Science); social bookmarks from CiteULike and Connotea; PLoS reader evaluation (i.e., readers’ feedback on the article in the form of comments, notes, and star ratings); and discussion of the article in blogs (2). These alternative metrics (or “altmetrics” as they are commonly known) for articles and even datasets and presentations are becoming easier to track via Web services such as Total-Impact (http://total-impact.org) and Altmetric (http://www.postgenomic.com), who offers explorer and browser-based bookmarklet applications.

Journal-level metrics

Journals are also assessed by different criteria. The impact factor, listed in Thomson Reuters’ Journal Citation Reports, assigns journals a numeric score based on the frequency with which the average article in the journal is cited over a set period of time (3). Whereas the impact factor tracks straight citations, the Eigenfactor score (http://www.eigenfactor.org) is derived from a formula based on citations from a journal over a 5-year period, with citations from highly ranked journals given more weight. Journal self-citations are not included in the Eigenfactor score, unlike the impact factor score.

A caveat of note for journal-level metrics: specialized journals or those published by societies may disseminate your work more efficiently to colleagues in your field than a “high-impact” general-interest science journal. Reaching your intended audience is the surest way to enhance the visibility and impact of your research.

Going beyond the metrics: the Becker model

It is tempting to use these metrics as an objective way to assign value or worth to a researcher’s output or to an individual journal. Although these metrics can be helpful in understanding research efforts, they cannot be evaluated in a vacuum. To understand the true impact of research, metrics derived from publication data must be supplemented with indicators that demonstrate tangible outcomes such as clinical implementation, benefit to the community, influence on legislation or policy, and economic benefit. Publication data alone do not provide a full narrative of research impact, nor are they predictive of meaningful health outcomes.

The Becker Medical Library Model for Assessment of Research Impact (4,5) serves as a framework to quantify and document research impact based on research outputs and activities. It includes resources for locating evidence of research impact and strategies for enhancing research impact. The site offers reporting templates, a glossary, and examples of relevant indicators of impact across the research process as well as a sample of a completed report. The use of publication data in tandem with the Becker Model provides a more robust overview of the impact of research to accomplish a host of higher-order activities that are critical in today’s biomedical research world, including the following: Justification of future requests for funding; quantification of return on research investment; discovery of how research findings are being used; promotion and tenure activities; identification of possible collaborators; demonstration that research findings are resulting in meaningful health outcomes; anddiscovery of community benefit as a result of research findings.

Notes

[1] Kristi L. Holmes and Cathy C. Sarli are affiliated with the Bernard Becker Medical Library, School of Medicine, Washington University, St. Louis.

References

1.Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 2005; 102:16569–16572.

2.Public Library of Science. Article-level metrics. http://article-level-metrics.plos.org.

3.Thomson Reuters. The Thomson Reuters impact factor. http://thomsonreuters.com/products_services/science/free/essays/impact_factor.

4.Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc 2010; 98:17–23.

5.Bernard Becker Medical Library. Assessing the impact of research. The Becker Medical Library Model for assessment of research impact. https://becker.wustl.edu/impact/assessment/index.html.

May 2012 (Vol. 4, Number 5)