The Academic Reputation Index is the centrepiece of the QS World University Rankings® carrying a weighting of 40%. It is an approach to international university evaluation that QS pioneered in 2004 and is the component that attracts the greatest interest and scrutiny. In concert with the Employer Reputation Index it is the aspect which sets this ranking most clearly apart from any other.
QS World University Rankings® are based in part on hard data and part on factors drawn from two large global surveys – one of academics and another of employers. These are a key characteristic of the QS ranking approach and offer some key benefits.
QS has rejected many proposed criteria (e.g. financial metrics like research income) which cannot be independently validated, or are subject to exchange rate and business cycle fluctuations. Instead, our Advisory Board favour maintaining a strong emphasis on peer review, for important reasons:
Source of Respondents
The results are based on the responses to a survey distributed worldwide academics from a number of different sources:
QS has been conducting this work since 2004 – all previous respondents to our survey are invited to respond again to provide us with an updated viewpoint on the quality of universities in their broad field. In 2014, 1,724 previous respondents returned to revise their response.
An academic publishing company headquartered in Singapore, World Scientific publishes about 500 titles a year as well as 120 journals in a variety of fields. World Scientific holds a subscription database well in excess of 300,000 worldwide from which, until 2010, QS drew 180,000 active records. The effectiveness of this channel had dropped off over the years and in 2011 QS chose to redirect and draw on more records from the Mardev lists. Responses from this channel will remain in the sample for at least two years and World Scientific may be drawn upon in the future to fill any specific shortfalls.
The data division of Reed Business Information, Mardev-DM2 is one of the world’s leading providers of business information and services. Mardev-DM2 controls access to IBIS (International Book Information Service), a database with over 1.2 million academic and library contacts. This channel has grown increasingly effective over the years and in 2014 QS drew 200,000 records.
In 2010, QS initiated an Academic Signup process to enable the thousands of interested academics we meet each year to actively signal their interest in participation. Volunteers are screened to ensure institutions are not using the signup process to unduly influence the position of their own or rival institutions. Over 25,000 academics have signed up since the process was launched in February 2010.
Institution Supplied Lists
Since 2007, institutions have been invited to submit lists of employers for us to invite to participate in the Employer Survey. In 2010, that invitation was extended to lists of academics also. Since academics are not able to submit in favour of their own institution, the risk of bias is minimal, nonetheless submissions are screened and sampling applied where any institution submits more than 400 records. In 2014, nearly 400 institutions supplied lists contributing over 190,000 additional academic contacts.
Wherever sampling is required, respondents are selected randomly with a focus on delivering a balanced sample by discipline and geography. Naturally, all databases carry a certain amount of noise and email invitations do get passed on. Responses are screened to remove inappropriate responses prior to analysis.
The survey has evolved since 2004 but largely follows the same general principles. Respondents are not asked to comment on the sciences if their expertise is in the arts. Respondents are not asked to comment on Europe if their knowledge is centred on Asia. The survey asks each respondent to specify their knowledge at the outset and then adapts based on their responses, the interactive list from which respondents are invited to select features only entries from their own region.
The survey is broken into the following sections:
The work is not done once the survey is designed and delivered. Once the responses are received a number of steps are taken to ensure the validity of the sample.
Five Year Aggregation
To boost the size and stability of the sample, QS combines responses from the last five years, where any respondent has responded more than once in the five year period, previous responses are discarded in favour of the latest numbers.
The survey samples contributing to this work have been growing substantially over the lifetime of the project, resulting in inherently more robust reputation measures. The decision has been taken to extend the window for both reputation measures to five years instead of three years previously, with responses from the earliest two years carrying a relative weight of 25% and 50% respectively.
Any online survey will receive a volume of test or speculative responses. QS runs an extensive filtering process to identify and discard responses of this nature.
It is well documented on the basis of other high-profile surveys in higher education that universities are not above attempting to get respondents to answer in a certain fashion. QS run a number of processes to screen for any manipulation of survey responses. If evidence is found to suggest any institution has attempted to overtly influence their performance, any responses acquired through sources 4 and 5 (above) are discarded.