What makes a journal valuable?

What makes a journal valuable?

Posted by Kevin Smith in : Scholarly Publishing , trackback

For
almost 90 years, librarians, faculty authors, tenure review committees
and publishers themselves have relied on a single measure – the impact
factor – to determine the relative quality of different scholarly
journals. Impact factors are based on the number of times articles from a particular journal are cited in other scholarly articles. The citations to articles in one journal are cumulated to calculate the impact factor. It is fairly obvious that this system has some problems, however. For
one thing, frequency of citation is a poor marker for quality, since
all cited references to a work may not be positive and approving. To
posit an extreme example, many articles that cite one specific article
as a particularly bad example will boost the citation rate for that
article and could raise the impact factor of the journal that published
the flawed study. Also, journals are not all of equal
quality or influence (which is the point, after all), so many citation
from peripheral journals may not be as important as one or two
citations in the really influential and universally-read publications. Impact
factor can flatten these distinctions in regard to a single article,
although cumulation over time should cause the “best” journals to rise
to the top.

A new measure of journal quality, called the Eigenfactor,
tries to address this last problem by starting with an evaluation of
journal quality and assessing article impact on that basis. As their explanation of their methods says,

“Eigenfactor provides a measure of the total influence that a journal

provides, rather than a measure of influence per article… To make our

results comparable to impact factor, we need to divide the journal

influence by the number of articles published.”

Leaving aside the complex mathematics explained at
their site, the Eigenfactor is based on an algorithm that maps how a
hypothetical researcher would move from article to article based on
cited references. This mapping yields a measure of the amount of time that researcher would spend with each particular journal. The
score of a journal is based on that finding, and the influence of
articles is measured by the influence of the journal in which they are
published. This method corrects for peripheral citations and, it is claimed, for different citation patterns in different disciplines.

Both of these methods, however, measure the
quality of journals only from within the relatively closed world of
traditional periodical publication. Can we imagine ways
of assess journal quality that can account for external factors and
hence for the changes that are occurring within scholarship?

The advent of online aggregators of journal
content has offered one relatively simple external measurement of
journal impact which librarians have been quick to embrace – cost per
article download. It used to be very cumbersome to try
and tally which print journals were most used in a library, based on
how often copies were picked up and reshelved. Now
databases offer constantly updated counts of downloads which are easily
divided into the cost of the database to provide a measure of where
collections budgets are best spent. Since many downloads
will reduce the cost per download, this metric also can serve as a
rough indication of quality, or, at least, influence.

The real question I have, however, is how
to assess the importance of traditional journal publication vis-à-vis
newer, informal means of communication that are growing in importance
amongst scholars. As blogs, wikis and exchanges of
working papers via e-mail grow, scholars are getting their inputs and
influences from new sources, and web publication of various kinds often
supplements, and occasionally supplants, tradition publication. As the ACRL’s recent paper on “Establishing a Research Agenda for Scholarly Communications” puts it,

“Extant measures may suffer from being tightly coupled to traditional

processes while also inhibiting the application of other measures of

value. In the new digital environment, activities other than traditional

or formal publication should be valued in the reward structure for scholarship.”

I know of no metric that can yet account for the variety of informal publications and their relative influence. That, of course, is why it is part of a research agenda. As
these informal, digital means of sharing scholarly work become more
common, one of the principle functions of traditional publication –
that of communicating the finished products of research – may become
less and less important. Other functions, such as
registration, certification and preservation, may continue to rely on
traditional journals for a longer time. But the academic
world needs to look carefully for ways to evaluate and compare the
influence of a variety of new communications if it is to value
scholarship based on its true impact.

Powered by ScribeFire.