Thomson's Rebuttal Recently, there was a very strong editorial in the Journal of Cell Biology (doi: 10.1083/jcb.200711140) that made some claims against Thomson Scientific's citation products: JCR and Web of Science. Thomson came out with a point by point rebuttal and explanation (http://scientific.thomson.com/citationimpactforum/8427045/).

I think this is a very healthy discussion. Many people would like there to be one perfectly clean, justifiable number we can use to rank people, journals, articles, institutions, etc. Wouldn't it be easier to allocate funding if you could sum a person's history and potential up with a single number or rank order? But we know that this isn't how the world works - it's not how people work (either individually or socially), nor is it how we should judge people. There are scientifically valid things you can do with citations to map research, look at connections between {articles, journals, people, institutions, countries, patents}, and look at relative citedness of articles or author's work or journals... but you can't sum up a person, institution, journal, etc., with a single number.

This citation analysis stuff is really hard to get your mind around and it seems so simple and obvious. Statements made recently in ACM publications, JASIST, ArXiv, on library listservs, and elsewhere show profound misunderstandings of how citations work, how Thomson products work now, and how they were originally designed (including trade-offs thoughtfully made). Many of these statements attribute malice or incompetence where a more pragmatic approach could be useful.

I think anyone who's played with citation analysis has been frustrated with data quality problems. In my tiny bit of experience, WoS data are way, way, way more clean than anything else, but they, too, are dirty. It turns out that people make mistakes, that things are cited incorrectly, that there are inefficiencies in the system. Garfield and de Solla Price and many others following them have clearly explained why certain choices were made and what the impact would be. Some of these choices have led to a western bias in WoS. Not terribly important until recently (next 5-10 years this will become a deal breaker). Some of these choices have also made WoS inadequate for CS and many areas of engineering in which conference papers are as important as journal articles.

I would recommend anyone who has a serious interest in this stuff subscribe to the SIGMETRICS mailing list. Also, for more on this particular issue, see: Leydesdorff, L. (in press) Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science. DOI: 10.1002/asi.20743

Update: Looks like the Leydesdorff article has a real citation: Journal of the American Society for Information Science and Technology, Volume 59, Issue 2 (p 278-287) . It really is a must read for people remotely interested in science policy, citation analysis, research evaluation...  ¶ 9:41 AM| |cites (technorati) |