| 40% |
QS World University Rankings® are based in part on hard data and part on factors drawn from two large global surveys – one of academics and another of employers. These are a key characteristic of the QS ranking approach and offer some key benefits.
QS has rejected many proposed criteria (e.g. financial metrics like research income) which cannot be independently validated, or are subject to exchange rate and business cycle fluctuations. Instead, our Advisory Board favour maintaining a strong emphasis on peer review, for important reasons:
Geographical/Cultural Diversity – Many evaluations seem based on a US model of what defines excellence in a university. Thus their results are often dominated by English-speaking, comprehensive, large universities with medical schools. A widely distributed pool of academic experts help identify excellence in areas unmapped by other metrics, resulting in institutions from 32 countries appearing in the top 200 in QS’ ranking.
Unbiased approach to different subjects – Without peer review, institutions with key strengths in Arts and Social Sciences might be penalised in the rankings simply because they don’t publish much research.
Contemporary Relevance – Founded as recently as 1991, HKUST came top in the QS Asian University Rankings in 2011. Nanyang Technological University was also formed in 1991, through merger, and is the top rated university in Asia within the classification of large, multidisciplinary, research intensive institutions without a medical school.
Reduced Language Bias – Respondents to our academic survey identify with research excellence both in English and their native languages, which avoids a bias towards internationally recognised journals published in English.
Statistical Validity – Over 15,000 academic respondents contributed to our 2010 academic results, returning over 120,000 individual statistical observations. Independent academic reviews have confirmed these results to be more than 99% reliable.
Resistant to Data Manipulation – The peer review survey results are collected independently and in such numbers so as to become almost impossible to manipulate and very difficult for institutions to ‘game’.
Academic Reputation Survey
The Academic Reputation Index is the centrepiece of the QS World University Rankings® carrying a weighting of 40%. It is an approach to international university evaluation that QS pioneered in 2004 and is the component that attracts the greatest interest and scrutiny. In concert with the Employer Reputation Index it is the aspect which sets this ranking most clearly apart from any other.
Source of Respondents
The results are based on the responses to a survey distributed worldwide academics from a number of different sources:
- Previous Respondents
QS has been conducting this work since 2004 – all previous respondents to our survey are invited to respond again to provide us with an updated viewpoint on the quality of universities in their broad field. In 2011, 3,633 previous respondents returned to revise their response. - World Scientific (www.worldscientific.com)
An academic publishing company headquartered in Singapore, World Scientific publishes about 500 titles a year as well as 120 journals in a variety of fields. World Scientific holds a subscription database well in excess of 300,000 worldwide from which, until 2010, QS drew 180,000 active records. The effectiveness of this channel had dropped off over the years and in 2011 QS chose to redirect and draw on more records from the Mardev lists. Responses from this channel will remain in the sample for at least two years and World Scientific may be drawn upon in the future to fill any specific shortfalls. - Mardev-DM2 (www.mardev.com)
The data division of Reed Business Information, Mardev-DM2 is one of the world’s leading providers of business information and services. Mardev-DM2 controls access to IBIS (International Book Information Service), a database with over 1.2 million academic and library contacts. This channel has grown increasingly effective over the years and in 2011 QS drew 200,000 records. - Academic Signup
In 2010, QS initiated an Academic Signup process to enable the thousands of interested academics we meet each year to actively signal their interest in participation. Volunteers are screened to ensure institutions are not using the signup process to unduly influence the position of their own or rival institutions. Over 2,700 academics have signed up since the process was launched in February 2010. - Institution Supplied Lists
Since 2007, institutions have been invited to submit lists of employers for us to invite to participate in the Employer Survey. In 2010, that invitation was extended to lists of academics also. Since academics are not able to submit in favour of their own institution, the risk of bias is minimal, nonetheless submissions are screened and sampling applied where any institution submits more than 400 records. In 2011, over 200 institutions supplied lists contributing over 60,000 additional academic contacts.
Wherever sampling is required, respondents are selected randomly with a focus on delivering a balanced sample by discipline and geography. Naturally, all databases carry a certain amount of noise and email invitations do get passed on. Responses are screened to remove inappropriate responses prior to analysis.
The Survey
The survey has evolved since 2004 but largely follows the same general principles. Respondents are not asked to comment on the sciences if their expertise is in the arts. Respondents are not asked to comment on Europe if their knowledge is centred on Asia. The survey asks each respondent to specify their knowledge at the outset and then adapts based on their responses, the interactive list from which respondents are invited to select features only entries from their own region.
The survey is broken into the following sections:
- Personal Details
Name, Institution, Job Title & Classification, Department, Years in Academia - Knowledge Specification
Country – respondents are requested to indicate which country they have most familiarity with rather than the country where they are based. This enables new international faculty members to comment on their sphere of knowledge rather than speculate on an area they may yet know little about.Region – regional knowledge responses are grouped into three supersets that define the list of institutions from which the respondent can select, these are Americas; Asia, Australia & New Zealand; and Europe, Middle East & AfricaFaculty Area – respondents are asked to select one or more faculty areas in which they consider their expertise to lie. These are Arts & Humanities; Engineering & Technology; Life Sciences & Medicine; Natural Sciences; and Social Sciences. Sections 3 and 4 below are repeated for each faculty area selected.Field – respondents are asked to select up to two specific fields that best define their academic expertise - Top Domestic Institutions
Respondents are asked to identify up to ten domestic institutions they consider best for research in each of the faculty areas selected in Section 2. Their own institution, if it would otherwise be included, is excluded from the presented list. - Top International Institutions
Respondents are asked to identify up to thirty international institutions they consider best for research in each of the faculty areas selected in Section 2. Their own institution, if it would otherwise be included, is excluded from the presented list. The list consists solely of institutions from the region(s) with which they express familiarity in section 2. - Additional Information
We use this section to gather additional information from respondents, such as feedback on previous publications and the importance of various measures in evaluating universities.
Response Processing
The work is not done once the survey is designed and delivered. Once the responses are received a number of steps are taken to ensure the validity of the sample.
Three Year Aggregation
To boost the size and stability of the sample, QS combines responses from the last three years, where any respondent has responded more than once in the three year period, previous responses are discarded in favour of the latest numbers.
Junk Filtering
Any online survey will receive a volume of test or speculative responses. QS runs an extensive filtering process to identify and discard responses of this nature.
Anomaly Testing
It is well documented on the basis of other high-profile surveys in higher education that universities are not above attempting to get respondents to answer in a certain fashion. QS run a number of processes to screen for any manipulation of survey responses. If evidence is found to suggest any institution has attempted to overtly influence their performance, any responses acquired through sources 4 and 5 (above) are discarded.
Results Analysis
Once the responses have all been processed, the fun really begins and it works as follows:
- For each of our five subject areas…
- Devise weightings based on the regions with which respondents consider themselves familiar – weightings are (now) based only on completed responses for the given question. This is slightly complicated by the fact that respondents are able to relate to more than one region.
- Derive a weighted count of international respondents in favour of each institution ensuring any self-references are excluded
- Derive a count of domestic respondents in favour of each institution adjusted against the number of institutions available for selection in that country and the total response from that country ensuring any self-references are excluded
- Apply a straight scaling to each of these to achieve a score out of 100
- Combine the two scores with a weighting 85% international, 15% domestic – these numbers were based on analysis of responses received before we separated the domestic and international responses three years ago, but a low weighting for domestic also reflects the fact that this is a world university ranking. We use 70:30 for the employer review.
- Square root the result – we do this to draw in the outliers but to a lesser degree than other methods might achieve – our intention is that excellence in one of our five areas should have an influence, but not too much of influence
- Scale the rooted score to present a score out of 100 for the given faculty area
- Combine the five totals with equal weighting to result in a final score which will then be standardized relative to the sample of institutions being used in any given context
KEY OBSERVATIONS
The process outlined above has a number of important implications. The steps (a) and (b) ensure that no single region is given greater emphasis over another and the steps (c) and (d) serve to ensure that high level of response from any single country do not systematically benefit all the institutions from that country.
Nobody who knows about education in my country (Argentina) can agree with this ranking. It’s impposible that the Universidad Católica de Buenos Aires or the Universidad Austral ranks better than the Universidad Nacional de Córdoba or the Universidad Nacional de La Plata.
It’s obviuos that the peopole who made this ranking didn’t understand the differenci between public and private education in the underdeveloped countries.
Please, be more serious…
We understand the difference between public and private institutions wherever they are in the world. Private institutions certainly tend to be less subject to central regulation and standards control which means that they can, in some cases, be very poor. However, some of the world’s leading institutions are private so being public is not a fundamental badge of superiority over private institutions. Against the measures we have selected and collected data for these are the results and they cannot match everyone’s previously held knowledge or perception. We will be continuing to maintain an open dialogue with institutions in the region over the coming months and hope to be able to include additional indicators in the future that may be bring more balance to the results.
Lack of criteria, 40% of reputation is very subjective, What is the role of a university? Why is a university important for a country? for the students, and for the development and prosperity of the countries? Number of students, facilities, research, contribution to increasing the PIB of the country, You have to show facts, so that the readers can believed and accept your classification.
Perhaps unsurprisingly, we get a lot of comments about Academic Reputation being very subjective. Each individual survey response is indeed subjective, a respondent will talk about what they know and not mention anything where there are gaps in their knowledge, but when you combine almost 34,000 responses the gaps in subjectivity cancel each other out. The truth is any measure is subjective, simply down to its selection and the weight it carries. Take papers and citations for example (used in almost every international study), their are different publication, reading and citing habits by faculty area, specific discipline, institution, city, country and region. Academics will generally cite other papers in the journals that they read – these may not be the best papers in each individual case but when everything is drawn together these anomalies are smoothed out.
I am writing from Greece. I am a Biologist from Athens University, which ranks in position 387. Un/ty of Crete ranks 451-500, along with Aristotle Un/ty, while National Technical Un/ty of Athens ranks 551-600 along with Un/ty of Patras.
This is almost impossible. National Technical Un/ty of Athens is widely considered as the first among those, in terms of research, funding, projects, and how seriously they take themselves there as academics.
I don’t understand how this came upside-down in your ranking. Maybe you should enrichen your criteria with more measureable facts. I understand this is difficult but it’s a serious matter as well.
However, looking at your criteria, my only dissagreement would be the 10% of employers’ opinion. Im’not confιdent that it’s a purely academic criterium, because it doesn’t measure scientific skills, but skills that make one favoured for labour. And one’s employer does not necessarily need a good scientist to work for them-the contrary is very often. Employers simply need employees to increase their profits, and cenrtainly have the specified technical skills-but that doesn’t make a good scientist.
Our rankings are designed to help prospective international students make better choices. The vast majority of these individuals are attending university to help build a marketable set of employability skills. In our view the employer reputation indicator is a critical aspect of what we do and perhaps one that deserves more emphasis.
It is the measurable facts we currently use that disadvantage the National Technical University of Athens – particulalry in faculty student ratio and our international measure – rather than the other way around.
Im currently writting a paper about this ranking and have a little question
Is there a difference between the The Times Higher Education–QS World University Rankings and the
QS World University Ranking? Find it a bit confusing.
During your collaboration you shared the same methodoligy?
During the publishing partnership between QS and Times Higher Education, QS produced the results and owned the intellectual propoerty over the methodology and the results. We have continued that work after Times Higher Education decided to go in a different direction. Every year we look at more institutions, gather more survey responses and make small refinements as circumstances demand but in broad terms our ranking today is much the same as it has been since 2005 when the Employer Reputation component was introduced for the first time.
thanks, i actually write a paper about your methodology in the context of my uni (university of st.gallen).
So the question is: do u have any print version online (pdf, word etc) of your methodology???, because its preferred as a source rather than a html webpage, that evolves over time…
The real detail of the methodology is only published here, but various summaries of it appear around the place. Our annual Top Universities Guide is probably the most formal publication that includes a summary of our methodology.
Who did you account for cultural differences across latin america? What is the number of responses per country and how it is weighted?
Cultural differences are extremely difficult if not impossible to adjust for scientifically. For example if we undertook a student survey globally, inviting respondents to rate their own institution on a 1-5 Likert scale and Chinese universities came in an average of 4 and British institutions an average of 2, what would that tell us? Perhaps some of the difference would be down to culture but how much exactly? The response levels by country are all available through the top menus on this site. Domestic and international responses are considered separately so a disproportionately high response from a given country offers no advantage to institutions in that country. This approach eliminates the need for a weighting by country.
Harvard can be very, very picky because of its reputation. Is Harvard the best institution of learning or has it simply attracted the best learners? Maybe it’s both, but you don’t know that.
Good question.
Perhaps over the entirety of history this is a virtuous circle. An instition that does something that sets it apart attracts good students (and faculty and staff) that enables it to further set itself apart. It was easier to achieve this when there were far fewer institutions in the world giving historic institutions an advantage.
However, this is not necessarily unfair. A university is principally a product of its people, and its facilities are a means to serve the needs of those people. A student attending Harvard, for example, may learn as much from the competition and interaction with class mates than from the fact that the campus has some old buildings and that research fellows in another faculty have access to first-rate apparatus.
“Over 15,000 academic respondents contributed to our 2010 academic results”:
15,000 respondents out of how many surveys did you send out? What is the response rate?
We pursue a lot of channels to collect academic responses. In 2011, alone, we got close to close to 24,000 academic responses and all channels combined probably meant in the region of 300,000 invitations were dispatched in total – so arounf an 8% response rate overall. That is, of course, an over simplification as some channels have higher and some lower rates of responsiveness. The channel with the least effective response rate in 2010 was dropped for 2011.
An 8% response rate? Eight percent? Can I quote you on that?
The surveys are conducted via an email invitation. Typical industry standard click-through rates to email campaigns tend to be in the region of 2-12%. Relevant links confirming this independently are:
http://emailstatcenter.com/ResponseMetrics.html
http://www.lyris.com/email-marketing/85-Average-Email-Click-Through-Rate
http://www.idealware.org/articles/email_metrics.php
Now a click through rate is only part of the story. Once someone clicks through, we also need them to actually complete the survey. Ultimately our survey response rates are strong whn compared against emnail led campaigns of similar nature so, sure, go right ahead and quote me, but make sure you do so in complete context.
I notice that there is a large group of universities with no score in this category. Do this means they did not get mentioned in any survey (or they got mentioned in a very small number of surveys)?
We only publish indicator scores for the top 300 institutions. A blank simply means the institution is outside that threshold.
QS World Rankings has indeed helped me to choose a University for my postgraduate studies in the field of Structural Engineering-Thank You very much. I am student of Civil Engineering from India and your rankings played a key role in selecting universities for me in the UK. And recent release of Civil and Structural ranking was just awesome.
I would like to ask one question, ‘Is this Civil and Structural ranking only for UG or both UG and PG?
Great to hear that our work has been able to help. It is stories like yours that keep us going.
As to your question… our discipline level rankings are based on three indicators – Academic Reputation, Employer Reputation and Citations per Paper. These are indicators of overall strength in the discipline and an institution that performs well will generally have world-class faculty and will offer good programs at both an undergraduate and a postgraduate level. We would strongly advise anyone looking at our results to carefully examine the course content and credentials of any institution they are considering before applying.
Thank you very much. I hope that you will upgrade your rankings more dynamically, so that rock solid decisions could be made based upon your rankings. It’s simply great to assess and rank the universities, that help students like me to choose universities to apply for.
All the best!
Thanks for the info on the 8% response rate. Actually, it’s very useful to have a comparison to put this into context.
That said, given the 92% non-response rate, and the removal of some ‘anomolies’, do you still think it wise to advocate Universities using these data as performance indicators?
I am more concerned by the response level and the resulting sample size than the response rate. Would I prefer us to have more responses to work with? Sure I would. Do I think that we can learn something meaningful from the opinions of almost 34,000 academics about the world’s universities? Absolutely. Do I think that, the survey approach at current response levels, has strengths that help overcome the shortfalls of other indicators? Definitely.
Do you advocate these rankings as performance indicators for Universities?
Your question seems to centre on whether we feel our rankings would serve as good performance indicators for a university.
This is a more complex question than it first appears. Regardless of methodology, stability and reliability – rankings of any sort – as they stand – do not make great performance indicators for a number of reasons:
1. The methodology is beyond your control and subject to change (even if it hasn’t changed for some time)
2. Rankings of any kind measure performance relative to other institutions and not to reliable and measurable milestones – you can do genuinely better but if the institutions around you are moving quicker, still you appear to have failed
3. Rankings are inherently limited in scope and range and fail to encapsulate all aspects that a university should consider to drive its collective performance
Our rankings are more stable than most, more reliable than most, more robust than most but they wouldn’t make good performance indicators without a layer of additional sophistication.
The best solutions I have seen are those where a university strips out the underlying metrics and establishes their own measurable and achievable targets for each – along with a selection of their own – and in some cases uses the popularity and simplicity of the ranking to help serve as a catalyst for establishing effective indicators internally.
It is obvious that no research can reflect the ideal situation but at least it gives more or less an idea about universities. I want to congratulate this research especially for one reason that it gave chance for new founded universities to get rid of some subjective criteria. Lets say new founded university has invested billion dollars for new laboratories and provide all opportunities for good education from one side and the old universities which had a great successes once upon a time but can not improve themselves due to some reasons. Briefly, there are some researches which gives great points for such criteria as to have Nobel laureates and so on which is not good criterion to reflect the present situation.
Best Regards.
I am curious as to whether the evaluation included both undergraduate and graduate studies or solely graduate. It seems to me that many of the top-ranking universities have stellar graduate programs but less than perfect undergraduate programs. Is one weighted more than the other?
If there has not been any differentiation between the two levels of study, I believe there should be, because many institutions have incredible graduate work being done but are not ideal for undergraduate education. If the rating “academic reputation” evaluates both undergrad and grad programs, there is a major flaw, because undergraduate institution reputation (which is generally based on how renowned the university is on a graduate, international scale) means nothing when the quality of undergraduate study is not as high.
Presently, academic reputation focuses on the research reputation of universities, which in many world contexts is also an effective proxy for the academic reputation of all of its programs. It is not necessarily an effective proxy for the academic quality of its programs though. The latter is a great challenge for all international ranking systems and one that we would be keen to be the first to overcome. We are currently conducting extensive student surveys which may help but other internationally transferable ideas are most welcome.