Methodology

Methodology refinements explanations

The methodology used to create the World University Rankings uses six robust measures which encapsulate the principal activities of global higher education. These measures are unchanged for the new 2015/16 Rankings. But as we explain here, the use we make of the data we collect has been improved markedly this year.

The first two of these measures involve asking informed people to identify the high points of the world university system. We do this by means of two annual surveys, one of active academics around the world, and one of recruiters. The academics are asked what their subject is and where the top 30 universities are in that field, although they tend to vote for a median of about 20. They cannot vote for their own institution. The employers are asked to name the subject or subjects in which they recruit graduates, and where they like to recruit them. These two measures account for 40 per cent and 10 per cent respectively of each institution’s possible score in this ranking.

These are the largest surveys of their kind, and this year have involved a total of 76,798 academics and 44,226 recruiters around the world who completed the survey in enough detail to provide valid and usable data. The academic survey covers all subjects, and the recruiter survey involves a wide range of public and private sector employers.

For 2015/16 we have improved the depth of these surveys by making more use of historic data. In the past we have counted the latest response from any one respondent within the previous three years. If you responded a year ago and two years ago, for example, only last year’s response would be used. We are still following this rule. But in addition, we are now using data which is four or five years old as well, weighting these votes at a half or a quarter respectively of more recent ones. Again, this material is only used if the same person has not also voted more recently. As well as adding stability to the ranking, this change improves its consistency. It means that we are using five years of data both for our surveys and for our citations measure described below.

The next measure we use is intended to find out whether universities have enough staff to teach their students. It is the ratio of faculty members to students, and accounts for 20 per cent of each institution’s possible score. This measure is unchanged from previous years.

Two indicators to which we apply a lower weighting are also unaltered from 2014, and indeed from the entire history of the QS rankings. Worth five per cent each of a university’s possible score, these are our measures of internationalisation, which we gauge on the basis of each university’s percentage of international faculty and of students. These measures show how serious a university is about being global. But in addition, they are an indirect indicator of quality. If a university is attracting staff and students from across the world, it is probably doing something right in terms of its research and teaching.

Rationalising citations

The biggest change to this year’s Rankings applies to the measure which makes up the final 20 per cent of each institution’s possible score. This is the measure of citations per academic faculty member. This indicator looks radically different this year because we have introduced a system to compensate for the large volume of citations generated by researchers in the life sciences and, to a lesser degree, those in the natural sciences. The need for this process, which we term normalisation, is apparent when one considers that the medical sciences account for 49 per cent of the citations in the Scopus database used in these rankings but only 14 per cent of university students (that figure being for the UK). By contrast, the arts and humanities make up nearly 30 per cent of students but only one per cent of citations, because of their very different publishing culture.

We believe that it is right to correct for this bias at the faculty level, in other words in terms of the arts and humanities; the social sciences, including management; the natural sciences; engineering and technology; and the biomedical sciences. We have normalised the weight of these five areas in our academic survey since its creation in 2004.

The normalisation process works by weighting the citations from each of these areas at 20 per cent of the total.

But even this reform does not recognise the full variation in academic publishing patterns around the world. In the arts and humanities and in the social sciences, a large amount of research is not published in English and does not appear in journals, reducing its chance of appearing in Scopus’s citations database. We allow for this by further adjusting the citations in these two areas, but not the other three, in accordance with the publishing pattern in each university’s home country, as reflected in the total percentage of papers in Scopus in these two fields.

Finally, the data we use will continue to cover five years of the Scopus database. But it will no longer credit citations where the paper has more than ten affiliated institutions. We feel that this thin level of participation is not worth acknowledging. This change cuts out only 0.34 per cent of Scopus papers.

by Martin Ince

 

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *