How we measure you up (a.k.a. "Vanderbilt?!")

The January 12, 2007 issue of the Chronicle of Higher Education ran a cover story about a new system of academic rankings, designed as an alternative to the established National Research Council rankings.  The article is behind a subscriber firewall, but I'll quote selected passages:

The Faculty Scholarly Productivity Index, partly financed by the State University of New York at Stony Brook and produced by Academic Analytics, a for-profit company, rates faculty members' scholarly output at nearly 7,300 doctoral programs around the country. It examines the number of book and journal articles published by each program's faculty, as well as journal citations, awards, honors, and grants received.
[...]
The index relies on Scopus, a database that compiles journal publication and citation data from more than 15,000 journals, while it counts books using Amazon.com, whose database matches the Library of Congress catalog.

The index incorporates grant data collected either from federal agencies directly or from information on their Web sites, including from the National Institutes of Health, the National Science Foundation, and the National Endowment for the Humanities, among others, though notably not the Department of Defense, since it does not release data on grants to individuals.

The company collects information on honors and awards from the Web sites of 55 organizations, including those that grant Nobel Prizes and MacArthur fellowships.

The variables are weighted differently depending on the program. For example, in fields such as chemical engineering, where books are not considered a crucial measure of productivity, they are not calculated as part of the rankings. And in some fields where honors and awards are not considered important, such as animal science, those are not included.
[...]
Charlotte V. Kuh, director of the [National Research] council's study [that is, the standard ranking system for universities, the one that put Berkeley at the top in 1995 and so forth], says its latest version relies less on reputations. "The measures of perceived quality are actually grounded in things that can be measured," she says, such as publications and citations. Other data will round out the picture, including information on graduation rates and how long it takes students to get their degrees.

That is a notable difference between the council's study and Academic Analytics' index. The former takes into account factors that directly influence graduate students, such as whether doctoral students get university-paid health-care benefits, whether teaching assistants have a collective-bargaining agreement, and the graduate student body's racial makeup. Both faculty members and graduate students were questioned for the council's latest study.

As for the financing delay, Ms. Kuh says, it couldn't be helped. The study was originally scheduled to be finished in 2005, but the data will not be available until the end of this year, with analytical essays and follow-up reports coming out in 2008.
[...]
[I]n physics, the council put Princeton at No. 2 and MIT at No. 3. Neither make Academic Analytics' top 10.

Lydia S. Snover, director of institutional research at MIT, explains that Academic Analytics' methodology may be part of the problem.

Because the company counts only faculty members listed under specific departments, it missed some scientists connected to MIT's numerous interdisciplinary research centers. Some of those scholars are the principal investigators on multimillion-dollar grants that other physics faculty members also work on, says Ms. Snover. But those grants and the scholars' publications did not get counted in the physics rankings.

Mr. Martin counters that faculty members who are purposely not listed on a program's Web site should not be counted, because if they were heavily involved with graduate students, an important criterion for being part of a graduate program, they would be listed.

The upshot: this is more useful as an internal tool for administrators curious about trends in their faculty's productivity than as a guide for the general public, prospective grad students, etc.  When I first heard about the rankings (via CUNY) and thought they were just counting articles, it seemed positively obscene: scholars who can produce large amounts of quality work are a pretty small minority in the humanities, and some brilliant people (and excellent mentors) often publish very few books and articles.  In fact, the possibility of an inverse relationship occurred to me.  In the sciences— MIT's skepticism aside— maybe it's different.  But a cult of productivity in literary studies just seems ridiculous, compared to the value of productivity in biomedical research, for instance, where research productivity translates into pharmaceutical and therapeutic productivity.  At this point I feel, for what it's worth, that teaching and interpersonal communication are the point, the goal, of literary studies.  There's really nothing to prove, unless you're Moretti; and if you think of productivity as a quintessentially capitalist virtue, then the fact that there's no money to speak of to be made from it— no market, no well-defined position in a network of exchanges— makes high productivity seem nonsensical.  But there you have CUNY's press release, which seemed to me to take pains to amplify the Chronicle's suggestion that Academic Analytics' system is an improvement on the NRC's, and the schools that benefit by it will, rationally enough, try to encourage the approach.

It's not bad to have a marketplace in higher education, don't get me wrong— the tyranny of prestige screws far more people than it helps.  If the comp lit students at Vanderbilt can get some mileage off these results— assuming its insanely goal-driven faculty hasn't caused all of them to drop out from lack of attention— that's great for them.  There's even a creepy possibility that the emphasis on productivity and "culling deadwood" will add liquidity to the job market, or reduce everyone to adjunct status.  (Note terrifying memo: "No more than a reasonable number of lecturers (e.g. four or five) should share one office."  The Black Hole of Cal State?)  But I keep looking for the object of productivity, its final cause, and seeing a void instead, and something about it really unsettles me.

Oh, and lastly: I had to wash my hands after touching the Chronicle.  Shudder.