It’s professional review time here in Academia and I’m engrossed in my now annual routine of pondering the meaning of the statistics we all use to justify ourselves. Enter Campbell’s Law:
“The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Campbell’s original paper can be found here. It’s a similar concept to Goodhart’s law:
“When a measure becomes a target, it ceases to be a good measure.”
Basically, the more we emphasize a particular statistic as an indicator of merit, the less effective it becomes. An example is the en-vogue topic of using Journal Impact Factor (JIF) as a measure of publication performance. Chelsea has a great historical blog post on the origination and growth of JIF in academia. In short, it was invented in the 1950’s to help librarians efficiently purchase the most read journals, as determined by the ratio of citations to publications of the journal. It says nothing about a particular publication in the journal, has serious outlier problems, and has remained unchanged despite the entirely different landscape for finding and reading publications we now have. Take the egregious case of Acta Crystallographica:
“The effect of outliers can be seen in the case of the article “A short history of SHELX”, which included this sentence: “This paper could serve as a general literature citation when one or more of the open-source SHELX programs (and the Bruker AXS version SHELXTL) are employed in the course of a crystal-structure determination”. This article received more than 6,600 citations. As a consequence, the impact factor of the journal Acta Crystallographica Section A rose from 2.051 in 2008 to 49.926 in 2009, more than Nature (at 31.434) and Science (at 28.103). The second-most cited article in Acta Crystallographica Section A in 2008 only had 28 citations.“
While this is a somewhat honest incident of statistical augmentation, corruption exists. I was shocked when the editor of one of our recent publications requested that “at least 5 articles from this journal should be added to your list of references prior to publication.” Our publication was the first, to our knowledge, ever on the specific topic and we had to work hard to find relevant publications from that journal. After talking with others, it’s not that uncommmon of a request. A sad demonstration that Campbell’s law is indeed a real issue.
The major problem with easy, ill-defined metrics like JIF is that they can grow without bound, i.e. they do not have balancing and restoring “feed-back” loops to keep the system in-check and are subject to rampant inflation. This is very much related to the 2nd law of thermodynamics and the concept of entropy. In physical (real) systems we thankfully never see growth without bound–entropy requires that something always breaks, some limit is reached, before things get too carried away. Problems sadly do occur when our made-up systems and metrics to guide society detach themselves from the reality of physical restoration loops. Dutch Tulip Mania is a tragic example and in some ways transferable to our current metric frenzy.
Yet the metrics remain and grow in use. Like my good friend says, “I use whatever metrics help me look the best.” Just like the bears asking for a handout, it’s not necessarily our fault that we’re rewarded for tailoring the presentation of our statistics, it’s our current unbalanced legalistic-performance meme shift in the US society. Thankfully we now have a human element in MME’s performance metrics which will hopefully add some balancing feedback loops to the process.