In the late 1990s HEFCE commissioned a study of university outcome metrics, maybe this was as an exercise to see if this could be used as an alternative to the more costly Research Assessment Exercises, or possibly produced as part of the process that led to the Dearing Report [De97].  I’ve not been able to trace this study (if you have any information please let me know), but from memory the spreadsheet of results had columns for three output metrics: total number of publications, total number of successful PhDs, and industrial income.  The last of these was considered ‘output’ as it was effectively a measure of esteem/respect by industrial partners.  These three measures were combined into a single research outcome metric. When ordered using this metric, Oxford, Cambridge and other unsurprising universities came at the top of the league table.

However, the spreadsheet had a column for the funding provided by HEFCE and a ‘value for money’ column that divided the research outcome metric by the money input.  If ordered by this value-for-money metric, a different story emerged.  This time Oxford and Cambridge were at the bottom of the league table, and the winner, the university with the best research value for money, was Northampton.

Of course, the metrics used were crude and it could be argued that not all publications, nor all PhDs are equally good and that with a metric that more clearly represented quality outcomes, the value-for-money table might look very different.  Certainly, maybe because it did not give the ‘right’ answers, it was quickly forgotten!

Roll on more than fifteen years and I had been a member of the REF 2021 computing sub-panel.  In January 2015, after the exercise was complete, the REF organisation ran a number of feedback sessions for panel members that included focus-groups facilitated by the RANK Organisation.  The event that I attended had several hundred attendees and the day ended with a panel session where the panellists included some overall REF Panel chairs, heads of funding bodies, etc.

Something in what one of the panellists said reminded me of the 1990s report, and so during the Q&A session I asked whether there were any plans to revisit this value-for-money exercise, but using the more reliable REF metrics.  The answer was “No”  – this was not so much a direct refusal, and without any arguments against doing it, more complete disinterest in the issue.

After the results were out the Metrics Tide report [WA15] was published using the detailed scoring data (which was then destroyed) and I did my own analysis of UoA11 using the publicly available data.  As REF assessors we had access to data about the grant income of each of the institutional UoAs we were assessing, and this was used as part of the assessment of research environment (i.e. as a form of output measure).  Nearly all the raw data that we had as assessors (but not the scores) was made available in the public domain after the exercise was complete, but one notable absence was this financial data.  I’m not sure of the reason for this, possibly this was because the data was obtained from HESA, which charges for data retrieval, and REF had not paid for public use.  It would have been possible to do a HESA request for the data, but it would have been costly, so I was not able to supplement my bias analysis with a value for money one [Dx15a, Dx15b, Dx16].

Roll on another ten years and while preparing a session recommending that people make use of the rich submission information available on the REF website, I realised that the REF2021 public domain data includes grant funding income.  This has now made it possible to perform independent value-for-money assessment based on public domain data.