In the late 1990s HEFCE commissioned a study of university outcome metrics, maybe this was as an exercise to see if this could be used as an alternative to the more costly Research Assessment Exercises, or possibly produced as part of the process that led to the Dearing Report [De97].\u00a0 I\u2019ve not been able to trace this study (if you have any information please let me know), but from memory the spreadsheet of results had columns for three output metrics: total number of publications, total number of successful PhDs, and industrial income.\u00a0 The last of these was considered \u2018output\u2019 as it was effectively a measure of esteem\/respect by industrial partners.\u00a0 These three measures were combined into a single research outcome metric. When ordered using this metric, Oxford, Cambridge and other unsurprising universities came at the top of the league table.<\/p>\n
However, the spreadsheet had a column for the funding provided by HEFCE and a \u2018value for money\u2019 column that divided the research outcome metric by the money input.\u00a0 If ordered by this value-for-money metric, a different story emerged.\u00a0 This time Oxford and Cambridge were at the bottom of the league table, and the winner, the university with the best research value for money, was Northampton.<\/p>\n
Of course, the metrics used were crude and it could be argued that not all publications, nor all PhDs are equally good and that with a metric that more clearly represented quality outcomes, the value-for-money table might look very different.\u00a0 Certainly, maybe because it did not give the \u2018right\u2019 answers, it was quickly forgotten!<\/p>\n
Roll on more than fifteen years and I had been a member of the REF 2021 computing sub-panel.\u00a0 In January 2015, after the exercise was complete, the REF organisation ran a number of feedback sessions for panel members that included focus-groups facilitated by the RANK Organisation.\u00a0 The event that I attended had several hundred attendees and the day ended with a panel session where the panellists included some overall REF Panel chairs, heads of funding bodies, etc.<\/p>\n
Something in what one of the panellists said reminded me of the 1990s report, and so during the Q&A session I asked whether there were any plans to revisit this value-for-money exercise, but using the more reliable REF metrics.\u00a0 The answer was \u201cNo\u201d\u00a0 \u2013 this was not so much a direct refusal, and without any arguments against doing it, more complete disinterest in the issue.<\/p>\n
After the results were out the Metrics Tide report [WA15] was published using the detailed scoring data (which was then destroyed) and I did my own analysis of UoA11 using the publicly available data.\u00a0 As REF assessors we had access to data about the grant income of each of the institutional UoAs we were assessing, and this was used as part of the assessment of research environment (i.e. as a form of output measure).\u00a0 Nearly all the raw data that we had as assessors (but not the scores) was made available in the public domain after the exercise was complete, but one notable absence was this financial data.\u00a0 I\u2019m not sure of the reason for this, possibly this was because the data was obtained from HESA, which charges for data retrieval, and REF had not paid for public use.\u00a0 It would have been possible to do a HESA request for the data, but it would have been costly, so I was not able to supplement my bias analysis with a value for money one [Dx15a, Dx15b, Dx16].<\/p>\n
Roll on another ten years and while preparing a session recommending that people make use of the rich submission information available on the REF website, I realised that the REF2021 public domain data includes grant funding income.\u00a0 This has now made it possible to perform independent value-for-money assessment based on public domain data.<\/p>\n","protected":false},"excerpt":{"rendered":"
In the late 1990s HEFCE commissioned a study of university outcome metrics, maybe this was as an exercise to see if this could be used as an alternative to the more costly Research Assessment Exercises, or possibly produced as part … Continue reading