5.1  Factors that might affect the accuracy or trends of results

As noted there may have been implicit bias affecting REF results, but based on the 2014 outcomes if anything this would have acted to make the value for money more extreme.

There might also be differences across different areas of computing.  While there are few areas requiring large-scale equipment as there are in Physics, theoretical computing may require less resources than more application-oriented areas.  However, within computing the post-1992 institutions are more likely to have application-focused research, so again this would mean if anything that the underlying trend is even more extreme than is apparent in the REF data.

Academics have multiple demands on their time conflicting with research.  However, one would expect these time demands to be greater in teaching-intensive institutions, so one might therefore expect less efficient use of research time, but that is the opposite of the trend seen.  However, competing workloads may well be a factor at the level of individual institutions.

Similarly, one would expect the research environment in research-heavy institutions to be more conducive to effective use of research resources, but this is the opposite of what is seen in the data.

5.2  Alternative analyses

Funding comes for various purposes, some is focused on pure research (esp. UKRI), some for industrial projects, some for work with communities or local business, some for doctoral training, etc.  The different value-for-money measures all use the overall income figure and do not attempt to model the individual types of income and how they map to the three main REF criteria areas.  Given there are only around ninety institutions and many factors, a more complex analysis would struggle to avoid overfitting, but this could perhaps be attempted by pooling data from several UoAs with similar characteristics.

5.3  Individual institution results vs overall trends

I was Director of the Computational Foundry at Swansea University over a substantial part of the REF period, so it is somewhat embarrassing that Swansea sits low in the income–GPA curve in Figure 10.  However, I am also aware that during the period the Computational Foundry hosted Cherish DE a large network-style grant that registered as Swansea income but had widespread benefits across UK HE; in addition, student numbers grew far faster than staff during the REF period leading to one of the worst computing SSRs in the UK.  Without making excuses (!), there will be similar stories across every institution and so the data in these ‘league tables’ should not be read as producing an accurate value-for-money estimate for each institution in UoA11.  However, the overall trends across sectors of UK HE are likely to be robust.

Fig 10. Graph of overall GPA vs Total income for academic years 2013-14 to 2019-20
(Swansea University highlighted)

5.4  Possible reasons for the trends seen

Assuming these results are robust, why are we seeing the overall trend to greater effectiveness in lower-ranked institutions.  This is an important question as understanding this might allow improved research effectiveness everywhere.

Here are some possible reasons for the differing value for money across the sector, which may all apply to some extent.  Some are about the overall funding system:

• implicit bias within the funding system –  In principle, UKRI and similar finding tries to avoid institutional bias, but, as was evident from REF2014, it is hard to avoid the halo effect of a proposal from a ‘good’ institution.
• money follows money –  The presence of lots of money, lab resources, etc., may make funders feel that existing centres of expertise are a safer bet for further investment.  Indeed, the QR funding formula to some extent follows this pattern.

Both of the above effects could mean that weaker proposals from high-esteem institutions are more likely to obtain funding, thus dragging down their institutions’ net efficiency.

Some reasons are about potential problems within the high-esteem institutions:

• overcommitment of big hitters –  In many institutions a relatively small number of academics are responsible for a large proportion of grant funding.  This may lead to some projects not getting suitable levels of supervision/management.
• surface veneer of well-written grant applications – The collective knowledge and critical mass within a research-intensive institution may mean that a fundamentally less good ideas may be written in ways that appear to be stronger, leading to better funding success of lower quality research.

Others relate to strengths within the lower-esteem institutions:

• greater staff selection  –  Teaching intensive institutions with small numbers of research active staff have often made stronger distinctions between staff types, offering research active staff lower teaching and admin loads.  While this must be set against higher overall loads, this could still be a net benefit.  Of course, given changes in submission criteria post REF 2014, there are increasing numbers of teaching-only staff across the entire university sector.
• targeted support –Many universities have targeted internal funding and research support services. It is hard to compare the total volume of these, but where the university as a whole has a smaller number of research active staff accessing these internal funds and services, so there might be a higher net service level.
• greater expectations – Where external funding, especially ‘cherished’ sources, is rare even small grants are seen as special and often lead to outcomes well out of proportion to the monies received.
• greater investigator input –  Because funding levels are lower, this may mean that there is a greater proportion of investigator/academic time (whether or not properly costed!) dedicated to each pound of grant funding; the obverse of ‘overcommitment of big hitters’!

5.5  Potential policy implications

The obvious take-away is that a shift to rebalance funding across the sector is likely to lead to a greater overall volume of high-quality research.  The mechanisms for this however may not be easy as UKRI peer review is notoriously resistant to policy nudges.  Also this is not about a complete levelling, it is clear that there are centres of excellence producing a large volume of world-leading work, but this seems to carry with it a far less effective tail.

Structurally there may be value in encouraging more projects that cross between facets of the sector nurturing islands of research excellence across the whole of HE and leveraging the greater efficiency of those pockets.  Of course, any such initiatives would need to ensure they leverage the best of both, rather than regress to the mean!  Devolved nations may be well placed to facilitate and encourage such cross-institutional connections.

Since the Thatcher years, the government pressure for centres of excellence has focused on creating critical mass.  In a digital age, these communities of practice need not be physically based within single institutions.  Again, positive pressure may be needed, as cross-institutional collaborations more often seem to be within pre- or post-1992 sectors rather than between them.

We clearly need a better understanding of the factors that make research spend more or less effective.  The suggestions in section 5.4 are based purely on the author’s personal experience and anecdotal evidence.  This suggests the need for richer studies of academic practice at a more detailed level so that funding policy can be rooted more solidly in evidence and data.  This need for ‘science of science policy’ was also one of the recommendations of the Metric Tide report [WA15].