Rankings are the public face of performance evaluation… but to geniunely evaluate and improve the performance of universities, much more actionable measures are required.
The lowdown on self-citations
Self-citations are often a point of discussion when it comes to ranking season, and never more than this year when, for the first time, we have had access to the necessary data to be able to remove self-citations from our consideration thanks to additional data supplied by Elsevier from SciVerse Scopus this year, but what does that really mean?
Simply put, a self-citation is an author citing their own previous work. Many leading academics may have every reason to do so having been responsible for some of the seminal works in their field, other academics may engineer opportunity to do so in order to influence their personal h-index or their institutional metrics. Either way, self-citations are a fact of life and can potentially be used contructively to trach threads of research and map academic careers.
As with many things there are a number of different ways the impact of self-citations can be viewed:
- Of all of the institutions we looked at the institution with the largest absolute number of self-citations, by some margin, is Harvard with over 93,000 representing 12.9% of their overall citations count
- The top five institutions producing over 3,000 papers, in terms of proportion of self-citations are all in Eastern Europe – St Petersburg State University, Czech Technical University, Warsaw University of Technology, Babes-Bolyai University and Lomonosov Moscow State University
- The top five in terms of the difference in citations per paper when self-citations are excluded are Caltech, Rockefeller, UC Santa Cruz, ENS Lyon and the University of Hawaii
- And the top 10 in terms of the difference in citations per faculty when self-citations are included are:
| # | Institution | Country |
|---|---|---|
| 1 | California Institute of Technology (Caltech) | United States |
| 2 | Rockefeller University | United States |
| 3 | Stanford University | United States |
| 4 | Gwangju Institute of Science and Technology (GIST) | South Korea |
| 5 | Karolinska Institute | United States |
| 6 | Princeton University | United States |
| 7 | Leiden University | Netherlands |
| 8 | Harvard University | United States |
| 9 | University of California, San Diego (UCSD) | United States |
| 10 | University of California, San Francisco (UCSF) | United States |
Caltech have been perennially number one, amongst institutions featured in the main ranking, for this indicator and nothing has changed this year, although due to the impact of removing self-citations the gap has closed somewhat in the underlying ratios and for some institutions further down the tables, this revision in the methodology may be felt more keenly.
2011 Employer Survey sign-up facility
The Employer Reputation Index is also based on a global online survey, but the target audience are employers. This indicator is unique amongst current international evaluations in taking into consideration the important component of employability. In principle, we are asking employers to comment on the quality of an institution’s graduates. We survey employers from all business sectors, including not for profit and public sector and regardless of company size.
If you are an employer and you would like to take part in the 2011 Employer Survey, please register your interest here.
2011 Academic Survey sign-up facility
The Academic Reputation Index is the aspect of the QS World University Rankings® towards which people’s greatest interest is directed. It is an approach to international university evaluation that, in concert with the Employer Reputation Index, is the defining feature which sets this ranking most clearly apart from any other.
The survey is distributed worldwide to academics from a number of different sources and the Academic Sign-up is one of them. Since the process was launched in February 2010, over 2,500 faculty members, university leaders and administrators volunteered to participate.
Today I would like to provide you with the opportunity to register your interest for 2011 here.
Influence of age on university “performance”
by Ben Sowter
The world is changing. And fast.
Higher education is no exception. In Saudi Arabia there are 28 universities, 22 of which were founded after the turn of the millenia. Economies worldwide are turning to the ever enticing notion of creating a “knowledge economy”. I read somewhere that we have generated more written content since 2003 than the in the whole of human history until that point.
In that environment – whilst rankings such as ours may treat all institutions equally – the reality is that date of establishment clearly has a part to play in the current success profile of universities. In broad terms, universities over 100 years old, and perhaps those over 50, have already reached their “terminal velocity” – the combination of reputation, government funding, scale of operation, organisational culture, international mix and alumni profile have reached a degree of equilibrium which makes radical shifts in performance – as measured by rankings or otherwise – exceedingly difficult to impose.
Funding and Governance
As a new year of global higher education rankings is due to commence (watch out for our Subject Rankings in April), our Research Manager and Rankings guru, Baerbel Ecklemann, offers her two cents and gives us an overview of the QS World University Rankings® and the QSIU team.
One question I am frequently confronted with is ‘How much does it cost to be part of your rankings exercises? My response ‘nothing’ usually generates an aura of disbelief and surprise, followed by ‘so, how does it work then?’
With this in mind, independent expertise, integrity and passionate commitment to the global higher education sector are at the very core of our work. So, how do we establish these principles?
I invite you to read on:
Autonomous operation or not?
by Baerbel Eckelmann
Autonomous operation or not?
One of the first challenges for any rankings project is to identify an initial list of institutions to evaluate. For feasibility reasons, it is extremely difficult to study every one of the 10,000 universities in the world. Policies and guidelines for the inclusion of institutions help us to keep the exercise manageable. Click here to read further. The fact is, certain institutions may feature in other evaluations but may be excluded entirely or partly from our study.
This is also true in the scenario of institutions campuses which want to be listed and hence treated as truly autonomous operations. In order to make an informed decision we have introduced the following qualifying questions: Continue Reading
The Bologna Process: Trends 2010 – A decade of change in European Higher Education
by Abby Chau
The Bologna Process is pressing on with its agenda of enhanced student mobility, standardisation of degrees and credit transfer, as well as quality assurance in order to promote institutional competition amongst its 46 participating countries. But as new countries contemplate membership, it is important to evaluate what the last ten years have achieved under this ambitious implementation programme.
The European University Association recently published Trends 2010 which examines a decade of higher education in the context of Bologna and outlines their goal for the future. Here are a few highlights, taken directly from the 100 –page report.
Overview
- Overall participation rates in higher education have increased by 25% on average between 1998 and 2006 – or as in Poland where enrolment increased by 90% during this period – albeit with significant differences across countries and across disciplines, with science and technology fields losing their attractiveness. (18)
- A recent study revealed that the number of 10-14 year olds in the EU is expected to fall by 15% between 2000 and 2020, resulting in a drastic reduction of the school-going population (Eurydice 2009), with a potential domino effect on higher education. The professoriate in higher education is greying and the ‘baby boom’ generation is going into retirement. Because these trends are uneven within a country (causing rural brain drain in some) and across Europe, they may lead to an exacerbated ‘brain war’ for students and academic staff, within Europe, at a time when the global competition for talents is heating up and international ranking schemes are proliferating and forcing institutional leaders to rethink their positioning within the global higher education community (19).
- The concept of academic freedom is changing – some say eroding – because academics are pressured to be successful in seeking funding for their research to match the research strategies and priorities of their institutions (22) Continue Reading
HE News Brief 4.5.10
by Abby Chau
A late post with news articles for week commencing May 4th:
- The pressure to do well on international league tables has caused a few UK universities to allegedly put pressure on students to fib on the National Student Survey. According to complaints logged with the HEFCE, students at Swansea, Anglia Ruskin, Derby, Leicester, Portsmouth, Sunderland, Kingston, and London Metropolitan were pressured by university lecturers and heads of department to score high marks on their university experience.
Full Story: Telegraph
More: The Guardian
- Education stocks dropped when a U.S Department of Education official compared for-profit institutions to Wall Street firms who caused the financial meltdown. Deputy Undersecretary of Education Robert Shireman said that not only is training at these institutions questionable and they deplete federal education funding, but oversight in accrediting these for-profits is dubious.
Full Story: Bloomberg Business Week
- Brazil will play an instrumental role in rebuilding Haiti’s devastated Higher Education system. According to a cooperation memorandum between the two countries, academic agreements will be discussed to promote internationalisation and scholarship programmes will be introduced for Haitians who plan on post-graduate study.
Full Story: iStockAnalyst
Continue Reading
AIR: A transferable model?
by Ben Sowter
Over the past two days I have had the pleasure of attending the 49th Annual Forum of the Association for Institutional Research in Atlanta. Normally at many of the conferences we attend there is a filtering process to find the right people to talk to, and then we still attract a fairly diverse audience from all levels of a university – the presentations we deliver have to meet these diverse needs. They frequently go well but are far less focused than this.
The very idea that there is a dedicated office in a university collecting, compiling, submitting and analyzing data on themselves, educating the strategy of the institution seems to be a predominately US inspired idea – it happens inrelatively isolated cases elsewhere – but that it has been running an annual conference for 49 years implies extraordinary foresight. Our experience has been of many institutions establishing these offices in response to the requirement to collect and submit data rather than for proprietary business intelligence needs.
The existence of this kind of an office within an institution provides a highly effective, expert conduit for our communication with universities, as has been the case with universities in many countries, but the practice taking broad enough hold to justify a national, regional or even global organisation and conference to support them provides the institutions with a great deal more leverage over third-party evaluating organisations such as ours. Indeed, Robert Morse, of the US News who kindly introduced us to this group has been coming to this event for 10 years – using it as a forum to collect qualified feedback and present and discuss proposed methodological changes. It will be an annual appointment on our calendar from this point forward.
If you work in an office of this nature, in a country where access to such a forum is not yet present, I would strongly recommend you investigate its foundation. Or join an international group.
Check out the US version on www.airweb.org or their international affiliates here www.airweb.org/page.asp?page=577
World University Classifications?
by Ben Sowter
I imagine this is too simple an idea to be particularly practical but would welcome feedback either way.
The THE-QS World University Rankings, amongst others, are frequently criticized in all sorts of ways, some fair and some not.
One of the most common observations is the failure of most aggregate ranking systems, whether international or domestic, to acknowledge the different missions and typologies of institutions.
In the case of the THE-QS exercise, large institutions are likely to be advantaged in terms or recognition whilst smaller ones may have greater ability to perform in some of the ratio based indicators.
In the US we frequently refer to the Carnegie classification system to better understand the nature of institutions that are featured in the rankings. What if we were to apply a similar, albeit simpler, concept to universities at a world level and include a classification alongside all ranking results.
Classifications might include:
Type A: Large, fully comprehensive
More than 10,000 students. Offer programs in all 5 of our broad faculty areas. Has a medical school.
(i) High Research - Over 5,000 papers in 5 year Scopus extract.
(ii) Moderate Research - 1,000-4,999 papers in 5 yyear Scopus extract
(iii) Low Research - 100-999 papers in 5 year Scopus extract
(iv) Negligible Research - Less than 100 papers in 5 year Scopus extract
Type B: Large, comprehensive
More than 10,000 students, operates programs in ALL of our 5 broad faculty areas. Has no medical school.
(i-iv) Reduced thresholds
Type C: Large, focused
More than 10,000 students. Operates programs in 3 or 4 of our broad faculty areas.
(i-iv) Reduced Thresholds
Type D: Large, specialist
More than 10,000 students. Operates programs in 1 or 2 of our broad faculty areas
(i-iv) Research thresholds set against mean or median for stated specialist faculty areas
Types E-H: same as above but for medium sized institutions. 4,000-10,000 students
Types H-K: Same as above but for small institutions – less than 4,000 students
A (u) or (p) could be added to denote institutions that only offer programs at either undergraduate or postgraduate level.
This is unlikely to, yet, be exhaustive but a system such as this may help readers put the ranking results in context. Thoughts and suggestions welcome.
← Older posts-
Recent articles
QS Best Student Cities 20126.2.12
The Leiden Ranking25.1.12
Count your saints: a new ranking criterion?25.1.12
