This article is from the source 'guardian' and was first published or seen on . It last changed over 40 days ago and won't be checked again for changes.
You can find the current article at its original source at http://www.theguardian.com/science/political-science/2015/jul/27/in-defence-of-the-ref
The article has changed 8 times. There is an RSS feed of changes available.
| Version 6 | Version 7 |
|---|---|
| In defence of the Research Excellence Framework | In defence of the Research Excellence Framework |
| (about 13 hours later) | |
| In academic circles, it’s become fashionable to denounce the Research Excellence Framework (REF) as the sector’s very own spawn of satan. The REF, a six-yearly assessment of the qualities and impacts of UK research, is used to allocate around £1.6 billion of public funding on an annual basis. In these austere times, you might think that would be enough to win it a few friends. Instead, the critics are lining up. | In academic circles, it’s become fashionable to denounce the Research Excellence Framework (REF) as the sector’s very own spawn of satan. The REF, a six-yearly assessment of the qualities and impacts of UK research, is used to allocate around £1.6 billion of public funding on an annual basis. In these austere times, you might think that would be enough to win it a few friends. Instead, the critics are lining up. |
| The REF is “a bloated boondoggle”, a “Frankenstein monster” and “a Minotaur that must be appeased by bloody sacrifices”. It is responsible for a “blackmail culture”, a “fever” and a “toxic miasma” which hangs over our campuses, choking the dying gasps of creativity from academic life. Entire books have been devoted to its insidious “hypocrisies”. Sir Paul Nurse and Lord Stern, the presidents of our most eminent academies, query its “wasteful and distorting” influence. Even Jo Johnson, the science minister, is now trying to sell the virtues of teaching assessment on the basis that he has “no intention of replicating the individual and institutional burdens of the REF”. | The REF is “a bloated boondoggle”, a “Frankenstein monster” and “a Minotaur that must be appeased by bloody sacrifices”. It is responsible for a “blackmail culture”, a “fever” and a “toxic miasma” which hangs over our campuses, choking the dying gasps of creativity from academic life. Entire books have been devoted to its insidious “hypocrisies”. Sir Paul Nurse and Lord Stern, the presidents of our most eminent academies, query its “wasteful and distorting” influence. Even Jo Johnson, the science minister, is now trying to sell the virtues of teaching assessment on the basis that he has “no intention of replicating the individual and institutional burdens of the REF”. |
| I was a REF sceptic once too. And I don’t pretend we can’t simplify and improve the exercise. But having spent the past year looking in detail at alternatives - particularly whether we could ditch the peer review, expert-led model of assessment that underpins the REF, in favour of a metric-based approach - let me out myself as a growing enthusiast. | I was a REF sceptic once too. And I don’t pretend we can’t simplify and improve the exercise. But having spent the past year looking in detail at alternatives - particularly whether we could ditch the peer review, expert-led model of assessment that underpins the REF, in favour of a metric-based approach - let me out myself as a growing enthusiast. |
| UK universities need the REF. We benefit enormously from the stable and relatively unrestricted funding that it delivers each year to our research system. And that system has been strengthened by the REF’s emphasis on the wider impacts of UK research, which are now captured in a searchable database of almost 7,000 case studies. | UK universities need the REF. We benefit enormously from the stable and relatively unrestricted funding that it delivers each year to our research system. And that system has been strengthened by the REF’s emphasis on the wider impacts of UK research, which are now captured in a searchable database of almost 7,000 case studies. |
| I’m prompted to write this by the full-throated attack on the REF launched last week by the free market think tank, the Institute of Economic Affairs (IEA). In a five-page briefing, the IEA calls for the REF to be abolished because it is too costly, distorts research priorities and can be gamed. | I’m prompted to write this by the full-throated attack on the REF launched last week by the free market think tank, the Institute of Economic Affairs (IEA). In a five-page briefing, the IEA calls for the REF to be abolished because it is too costly, distorts research priorities and can be gamed. |
| Instead the IEA wants quality-related (QR) funding, which is currently allocated through the REF, to be moved to the research councils. But for an economics outfit, the paper is surprisingly sloppy – it doesn’t even annualize these costs to reflect their spread across an entire assessment cycle. The £246 million full cost of the REF sounds like a lot if you set it against one year of QR funding. But across a six-year period, this is equivalent to 2.4% of the £10.2 billion of QR funding that is set to be allocated (or around £41 million per year). | Instead the IEA wants quality-related (QR) funding, which is currently allocated through the REF, to be moved to the research councils. But for an economics outfit, the paper is surprisingly sloppy – it doesn’t even annualize these costs to reflect their spread across an entire assessment cycle. The £246 million full cost of the REF sounds like a lot if you set it against one year of QR funding. But across a six-year period, this is equivalent to 2.4% of the £10.2 billion of QR funding that is set to be allocated (or around £41 million per year). |
| For the IEA to prove its point, these costs then need to be compared to the equivalent on the other side of the dual support funding system. Unlike HEFCE, which has conducted an admirably robust and transparent evaluation of REF2014, recent costs for the research council system are harder to obtain. The last detailed analysis was published in 2006. This estimated costs at around £196 million per year – so almost five times that of the REF. | For the IEA to prove its point, these costs then need to be compared to the equivalent on the other side of the dual support funding system. Unlike HEFCE, which has conducted an admirably robust and transparent evaluation of REF2014, recent costs for the research council system are harder to obtain. The last detailed analysis was published in 2006. This estimated costs at around £196 million per year – so almost five times that of the REF. |
| And bear in mind that in 2006, UK research funding was at near-record highs. Since then, we’ve had five years of flat cash funding, and across the research councils proposal success rates have dropped, in some cases to below 10 per cent. This inevitably increases the cost – and waste – in the system, as up to nine in ten applicants are investing large amounts of time and effort in unsuccessful proposals. | |
| I mention this not to criticise the research councils, who are doing the best job they can in a difficult funding climate, but simply to show the relative efficiency of allocating funding through QR and the REF. A healthy, vibrant research system needs diverse routes to funding, which is why the UK’s dual support system has served us so well over the past thirty years. | I mention this not to criticise the research councils, who are doing the best job they can in a difficult funding climate, but simply to show the relative efficiency of allocating funding through QR and the REF. A healthy, vibrant research system needs diverse routes to funding, which is why the UK’s dual support system has served us so well over the past thirty years. |
| Of course, as the IEA points out, the REF (like any evaluation system) has effects on research cultures, priorities and behaviours in universities. But these can be positive as well as negative, as discussed in our recent report The Metric Tide. And there are similar effects on the other side of the dual support system: damaging effects on research performance and morale, if success rates drop below 20 per cent, are well documented. | Of course, as the IEA points out, the REF (like any evaluation system) has effects on research cultures, priorities and behaviours in universities. But these can be positive as well as negative, as discussed in our recent report The Metric Tide. And there are similar effects on the other side of the dual support system: damaging effects on research performance and morale, if success rates drop below 20 per cent, are well documented. |
| If the REF were scrapped tomorrow, many of the evaluation and audit functions that have grown up around it would continue, as part of wider university management of research. Pressures for quality, impact, accountability and institutional benchmarking wouldn’t disappear. Indeed, one could argue that the REF acts as a bulwark against more harmful currents washing through higher education and research, such as university rankings and league tables. | If the REF were scrapped tomorrow, many of the evaluation and audit functions that have grown up around it would continue, as part of wider university management of research. Pressures for quality, impact, accountability and institutional benchmarking wouldn’t disappear. Indeed, one could argue that the REF acts as a bulwark against more harmful currents washing through higher education and research, such as university rankings and league tables. |
| So to make an evidence-informed judgement about the costs and benefits of the REF, we first need to compare it to the other side of the dual support system. We also need to reflect on the potential consequences of severe reductions or total abolition of QR funding, which would starve entire research fields (particularly in the social sciences, arts and humanities) of the long-term investment they need, and hit early career researchers the hardest. | So to make an evidence-informed judgement about the costs and benefits of the REF, we first need to compare it to the other side of the dual support system. We also need to reflect on the potential consequences of severe reductions or total abolition of QR funding, which would starve entire research fields (particularly in the social sciences, arts and humanities) of the long-term investment they need, and hit early career researchers the hardest. |
| In the 2015 spending review, the research community isn’t being offered a return ticket to some (mostly mythical) golden age in which public funding was doled out, without any demands for accountability. Instead, our already stressed funding system is facing another five years of flat cash – or worse – as the Department for Business, Innovation and Skills (BIS) grapples with George Osborne’s goal of up to 40 per cent more cuts. As Paul Jump, one of the most astute commentators on UK research policy, wrote recently in the Times Higher: | In the 2015 spending review, the research community isn’t being offered a return ticket to some (mostly mythical) golden age in which public funding was doled out, without any demands for accountability. Instead, our already stressed funding system is facing another five years of flat cash – or worse – as the Department for Business, Innovation and Skills (BIS) grapples with George Osborne’s goal of up to 40 per cent more cuts. As Paul Jump, one of the most astute commentators on UK research policy, wrote recently in the Times Higher: |
| I sometimes ask opponents of supposedly neoliberal excrescences such as the impact agenda and the research excellence framework how large a cut in the research budget they would be prepared to take in order to be rid of them....[It] does strike me as a very safe bet that research spending would be considerably lower – perhaps much more than 25 per cent lower - than it currently is if the academy were unable to demonstrate its quality and impact in terms that politicians and officials, for better or worse, can understand. | I sometimes ask opponents of supposedly neoliberal excrescences such as the impact agenda and the research excellence framework how large a cut in the research budget they would be prepared to take in order to be rid of them....[It] does strike me as a very safe bet that research spending would be considerably lower – perhaps much more than 25 per cent lower - than it currently is if the academy were unable to demonstrate its quality and impact in terms that politicians and officials, for better or worse, can understand. |
| If you read to the end of the IEA’s polemic, their real agenda becomes clear: faster, deeper and more painful cuts to the research budget. “There is a strong case,” the authors argue, “for reducing the total amount of government subsidy for research and expecting universities to generate their own funds...” Coverage of the IEA report in the Telegraph and Daily Mail complains about universities doing “too much research”, or research that is “pointless”. This is not a picture that those working in the UK’s universities will recognise. Nor would the IEA’s proposals ensure that the UK maintains its competitive edge as a research-intensive nation. | If you read to the end of the IEA’s polemic, their real agenda becomes clear: faster, deeper and more painful cuts to the research budget. “There is a strong case,” the authors argue, “for reducing the total amount of government subsidy for research and expecting universities to generate their own funds...” Coverage of the IEA report in the Telegraph and Daily Mail complains about universities doing “too much research”, or research that is “pointless”. This is not a picture that those working in the UK’s universities will recognise. Nor would the IEA’s proposals ensure that the UK maintains its competitive edge as a research-intensive nation. |
| Who cares what the IEA says about the REF? I worked in a think tank myself in the early 2000s, and even then the IEA occupied the crustier, swivel-eyed end of the market. However, one person who does appear to hold the IEA in high regard is Sajid Javid, secretary of state at BIS, who recently gave a keynote speech at the think tank’s 60th birthday celebrations. | Who cares what the IEA says about the REF? I worked in a think tank myself in the early 2000s, and even then the IEA occupied the crustier, swivel-eyed end of the market. However, one person who does appear to hold the IEA in high regard is Sajid Javid, secretary of state at BIS, who recently gave a keynote speech at the think tank’s 60th birthday celebrations. |
| George Osborne remains broadly supportive of science, but BIS’s role as a champion of the research system can no longer be assumed. In-year cuts of £150 million to HEFCE’s budget, announced last week, are a worrying signal of what may lie ahead in the autumn spending review. | George Osborne remains broadly supportive of science, but BIS’s role as a champion of the research system can no longer be assumed. In-year cuts of £150 million to HEFCE’s budget, announced last week, are a worrying signal of what may lie ahead in the autumn spending review. |
| Beyond the spending review itself, which got underway on 21 July, the Political Science blog can exclusively reveal that Sajid Javid has recently commissioned the consultancy McKinsey to undertake an “efficiency and effectiveness review” of all BIS-funded bodies, including the research councils, HEFCE and Innovate UK. | Beyond the spending review itself, which got underway on 21 July, the Political Science blog can exclusively reveal that Sajid Javid has recently commissioned the consultancy McKinsey to undertake an “efficiency and effectiveness review” of all BIS-funded bodies, including the research councils, HEFCE and Innovate UK. |
| This review, details of which have not been publicly announced, is clearly intended to shape BIS’s spending review submission to HM Treasury. Sajid Javid was apparently dismayed by the number of bodies that BIS is funding, so turned to McKinsey, which prides itself on a radical approach to cost cutting. | This review, details of which have not been publicly announced, is clearly intended to shape BIS’s spending review submission to HM Treasury. Sajid Javid was apparently dismayed by the number of bodies that BIS is funding, so turned to McKinsey, which prides itself on a radical approach to cost cutting. |
| The terms of reference for the McKinsey review, the evidence it will be based on, and its relationship to more public processes like the Nurse Review are unclear. A call to the BIS press office elicited no comment. My concern is that developments like this pose a further threat to QR funding, given the inherent bias inside HM Treasury against block funding allocations, particularly when expert peer review, rather than Whitehall edicts, control how the money is being spent. | The terms of reference for the McKinsey review, the evidence it will be based on, and its relationship to more public processes like the Nurse Review are unclear. A call to the BIS press office elicited no comment. My concern is that developments like this pose a further threat to QR funding, given the inherent bias inside HM Treasury against block funding allocations, particularly when expert peer review, rather than Whitehall edicts, control how the money is being spent. |
| I hope my fears are misplaced. The IEA says that research assessment promotes game playing. But next time you feel inclined to launch a broadside against the REF, do us all a favour, and bear in mind whose game you may in fact be playing. | I hope my fears are misplaced. The IEA says that research assessment promotes game playing. But next time you feel inclined to launch a broadside against the REF, do us all a favour, and bear in mind whose game you may in fact be playing. |
| James Wilsdon is professor of science and democracy in the Science Policy Research Unit (SPRU) at the University of Sussex (@jameswilsdon). He recently chaired an Independent Review of the Role of Metrics in Research Assessment and Management on behalf of HEFCE and BIS. | James Wilsdon is professor of science and democracy in the Science Policy Research Unit (SPRU) at the University of Sussex (@jameswilsdon). He recently chaired an Independent Review of the Role of Metrics in Research Assessment and Management on behalf of HEFCE and BIS. |