Lies, damn lies and statistics…

by Ben Sowter

 

It was Benjamin Disraeli that coined the phrase, “There are three kinds of lies: lies, damn lies and statistics” and many a commentator would have us accept that any ranking of universities falls under at least one of his three headings. Rankings of anything are certainly a complex operation, selection of indicators is delicate and subjective, assignment of weightings similarly so. One thing is for sure, ranking universities (or almost anything else of importance) is an inexact science but, whilst each has its own drawbacks, the majority of exercises out there have some basis in sound reason.

When looking at any ranking of institutions, it is crucial that the reader takes the time to figure out exactly what is being measured. This months revelation of another global ranking system of universities perhaps reveals the consequences of setting forth on a ranking design with a clear and specific agenda. In addition to the THE-QS, Shanghai Jiao Tong, HEEACT (Taiwan), Webometrics and Leiden (CWTS) exercises, a non profit organisation in Russia, known simply as RatER have recently released an additional ranking which was briefly to be found on www.globaluniversitiesranking.org.

At first glance the exercise looks compelling – the website is well laid out and simple to get around… but two crucial red flags are raised…

1. The Methodology

The methodology looks exceptionally detailed – using a far larger number of indicators than are used by any other ranking operating on a global scale. From our experience operating the THE-QS World University Rankings for five years, this level of detail is exceptionally difficult, perhaps impossible, to collect at that scale – especially given the provided list of institutions from which the ranking body claim to have been in contact. The only conclusion can be that many blanks have been left or arbitrary default values inserted.

2. The Results

Like most rankings the RatER exercise reveals some reasonably intuitive results – MIT tops their list, Harvard is at number 6… one place behind Moscow State University! It is not unlikely that other systems have been designed in such a way that leading Russian institutions may not feature highly – but the four other systems publishing an aggregate ranking Harvard is in the top 2, and in most it has been for some time. Moscow State? Not so much.

There are a very large number of universities in the world and even a considerable number of good universities. For its particular strengths I have little doubt that Moscow State is an excellent institution, but it is clear that this is a ranking that has been designed to dramatically inflate the performance of Russian institutions.

Whilst there may not be any such obvious evidence of a strong agenda in other rankings – this may serve as a caution to anyone placing too much stock in the results of ranking system – even our own. Virtually anything can be achieved with the manipulation of statistics – either deliberately or accidentally. For each individual stakeholder in a university different things are important – take what you need from rankings… and then back it up with your own research to make sure you’re choosing the right institution for you.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *