It's about trust, reliability, accuracy ... Stacy on Web4Lib reported that a vendor provided obviously bad usage statistics (reporting usages of databases she doesn't have access to) and when questioned, they said simply that it was a "known issue" and that they would *hide* those stats from her. ARGH! So she apparently tried a few different ways to tell them that hiding-omitting-scrubbing bad stats isn't really the answer she needs. Then they said "trust me" about the rest of the statistics!!! Wha?

She also had this problem with another vendor:

This isn't the only vendor that provided inaccurate usage data this year.
Another  vendor's statistics showed zero usage after our subscription
started. Since I had used it  at the beginning of  the subscription period,
it was clear something was wrong.  When this anomaly was reported, the
vendor never explained what the problem was, but sent us some
completely  different (and - surprise! - much higher) usage figures. 

So, we're supposed to be using more scientific ways of evaluating our collection (usage factors, impact factors, weighting systems...) and being able to justify costs based on use and need. In fact, some would like to completely remove the professional judgment of the librarian and use only usage metrics (not MPOW, btw, we're sensible). If this is the crap we allow from our vendors, there's no way we can do these evaluations systematically and have them have any relationship to reality. (I'm not criticizing Stacy -- she's obviously not allowing this but fighting it!)

This vendor in question is specifically listed in the COUNTER documents so should have completed an audit by a CPA. These audits are supposed to help with credibility, reliability, and validity. Maybe the higher ups at the company don't know what their help desk people are telling librarians who call?

Update 3/26/2007: Since this post, I have been in contact with Michael Gorrell, Senior Vice President, Chief Information Officer and Kate Hanson of EBSCO. They have been doing some in-depth troubleshooting and have found a problem in an algorithm that led to the errant usage report. See their support site for more information: http://support.ebsco.com/support_news/detail.php?id=341&t=h. As I said in a private e-mail to Mr. Gorrell- my intent was not to criticize EBSCO directly as they do seem to be cognizant of the importance (generally) and are working to fix the problem (ever since it got escalated from the original help desk person). Once again, libraries are very heavily reliant on vendor-provided statistics for decision making which is a bit scary. Also Mr. Gorrell corrected an assumption I made -- COUNTER audits are for content and formatting, not for accuracy.... sigh.

Labels: library management, metrics, usage statistics

  ¶ 1:20 PM| |cites (technorati) |