value for money in research – excellence or diversity

Government research funding policy in many countries, including the UK, has focused on centres of excellence, putting more funding into a few institutions and research groups who are creating the most valuable outputs.

Is this the best policy, and does evidence support it?

From “Big Science vs. Little Science: How Scientific Impact Scales with Funding”

I’m prompted to write as Leonel Morgado (Facebook, web) shared a link to a 2013 PLOS ONE paper “Big Science vs. Little Science: How Scientific Impact Scales with Funding” by Jean-Michel Fortin and David Currie.  The paper analyses work funded by Natural Sciences and Engineering Research Council of Canada (NSERC), and looked at size of grant vs. research outcomes.  The paper demonstrates diminishing returns: large grants produce more research outcomes than smaller grants, but less per dollar spend.  That is concentrating research funding appears to reduce the overall research output.

Of course, those obtaining research grants have all been through a highly competitive process, so the NSERC results may simply be a factor of the fact that we are already looking at the very top level of the research projects.

However, a report many years ago reinforces this story, and suggests it holds more broadly.

Sometime in the mid-late 1990s HEFCE the UK higher education funding agency, did a study where they ranked all universities against every simple research output metrics1. One of the outputs was the number of PhD completions and another was industrial research income (arguably whether an output!), but I forget the third.

Not surprisingly Oxford and Cambridge came top of the list when ranked by aggregate research output.

However, the speadsheet also included the amount of research money HEFCE paid into the university and a value-for-money column.

When ranked against value-for-money, the table was near reversed, with Oxford and Cambridge at the very bottom and Northampton University (not typically known as the peak of the university excellence ratings) was the top. That is HEFCE got more research output for pound spent at Northampton than anywhere else in the UK.

The UK REF2014 used an extensive and time-consuming peer-review mechanism to rank the research quality of each discipline in each UK university-level institution, on a 1* to 4* scale (4* being best). Funding is heavily ramped towards 4* (in England the weighting is 10:3:0:0 for 4*:3*:2*:1*). As part of the process, comprehensive funding information was produced for each unit of assessment (typically a department), including UK government income, European projects, charity and industrial funding.

So, we have an officially accepted assessment of research outcomes (that is government funds against it!), and also of the income that generated it.

At a public meeting following the 2014 exercise, I asked a senior person at HEFCE whether they planned to take the two and create a value for money metric, for example, the cost per 4* output.

There was a distinct lack of enthusiasm for the idea!

Furthermore, my analysis of REF measures vs citation metrics suggested that this very focused official funding model was further concentrated by an almost unbelievably extreme bias towards elite institutions in the grading: apparently equal work in terms of external metrics was ranked nearly an order of magnitude higher for ‘better’ institutions, leading to funding being around 2.5 times higher for some elite universities than objective measures would suggest.


From “REF Redux 4 – institutional effects“: ‘winners’ are those with 25% or more than metrics would estimate, ‘losers’ those with 25% or more less.

In summary, the implications both from Fortin and Currie’s PLOS ONE paper and from the 1990s HEFCE report suggest spreading funding more widely would increase overall research outcomes, but both official policy and implicit review bias do the opposite.

  1. I recall reading this, but it was before the days when I rolled everything over on my computer, so can’t find the exact reference. If anyone recalls the name of the report, or has a copy, I would be very grateful.[back]

ignorance or misinformation – the press and higher education

I guess I shouldn’t be surprised at poor reporting in the Mail, but it does feel slightly more serious than the other tabloids.  I should explain I have a copy of the Mail as it was the only UK paper when I got on the Malaysian Airlines plane in Kuala Lumpur on Tuesday evening, and it is the Monday copy as I assume it had flown out of the UK on the flight the day before!

Deepish inside, p22, the article was “UK students lose out in sciences” by Nick Mcdermott.  The article quotes a report by Civitas that shows that while the annual number of students in so called STEM (Science, Technology, Engineering and Maths) courses rose by around 6500 in the 10 years 1997-2007, in fact this is largely due to an increase of 12,308 in overseas students and a fall in UK students of nearly 6000.  Given an overall increase in student numbers of 600,000 in this period and employers “calling for more science graduates”, the STEM drop is particularly marked.

While the figures I assume are correct, the Mail article leaves the false impression that the overseas students are in some way taking places from the UK students, indeed the article’s title “UK students lose out” suggests precisely this.  I can’t work out if this is simply the writer’s ignorance of the UK higher education system, or deliberate misinformation — neither are good news for British journalism.

Of course, the truth is precisely the opposite.  Overseas students are not in competition with UK students for undergraduate places in STEM or other subjects, as the number of UK students is effectively controlled by a combination of Government quotas and falling student demand in STEM subjects.  The latter, a disinterest in the traditionally ‘hard’ subjects by University applicants, has led to the closure of several university science departments across the country.  Rather than competing with UK students, the presence of overseas students makes courses more likely to be viable and thus preserves the variety of education available for UK students.  Furthermore, the higher fees for overseas students compared with the combined student fees and government monies for UK students, means that, if anything, these overseas students subsidise their UK colleagues.

We should certainly be asking why it is that an increasing number of overseas students value the importance of a science/engineering training while their British counterparts eschew these areas.  However, the blame for the lack of UK engineering graduates does not lie with the overseas students, but closer to home.  Somehow in our school system and popular culture we have lost a sense of the value of a deep scientific education.  Until this changes and UK students begin to apply for these subjects, we cannot expect there to be more UK graduates.  In the mean time, we can only hope that there will be more overseas students coming to study in the UK and keep the scientific and engineering expertise of universities alive until our own country finally comes to its senses.

Qualification vs unlimited education

In “Adrift in Caledonia“, Nick Thorpe is in the Shetland Isles speaking to Stuart Hill (aka ‘Captain Calamity’).  Stuart says:

“What does qualification mean? … Grammatically, a qualification limits the meaning of a sentence. And that’s what qualifications seem to do to people. When you become a lawyer it becomes impossible to think of yourself outside that definition. The whole of the education system is designed to fit people into employment, into the system. It’s not designed to realise their full creativity.”

Now Stuart may be being slightly cynical and maybe the ‘whole of education system’ is not like that, but sadly the general thrust often seems so.

Indeed I recently tweeted a link to @fmeawad‘s post “Don’t be Shy to #fail” as it echoed my own long standing worries (see “abject failures“) that we have a system that encourages students to make early, virtually unchangeable, choices about academic or career choices, and then systematically tell them how badly they do at it. Instead the whole purpose of education should be to enable people to discover their strengths and their purposes and help them to excel in those things, which are close to their heart and build on their abilities.  And this may involve ‘failures’ along the way and may mean shifting areas and directions.

At a university level the very idea behind the name ‘university’ was the bringing together of disparate scholars.  In “The Rise and Progress of  Universities” (Chapter 2. What is a University?, 1854) John Henry Newman (Cardinal Newman, recently beatified) wrote:

“IF I were asked to describe as briefly and popularly as I could, what a University was, I should draw my answer from its ancient designation of a Studium Generale, or “School of Universal Learning.” This description implies the assemblage of strangers from all parts in one spot;—from all parts; else, how will you find professors and students for every department of knowledge? and in one spot; else, how can there be any school at all? Accordingly, in its simple and rudimental form, it is a school of knowledge of every kind, consisting of teachers and learners from every quarter. Many things are requisite to complete and satisfy the idea embodied in this description; but such as this a University seems to be in its essence, a place for the communication and circulation of thought, by means of personal intercourse, through a wide extent of country.”

Note the emphasis on having representatives of many fields of knowledge ‘in one spot’: the meeting and exchange, the flow across disciplines, and yet is this the experience of many students?  In the Scottish university system, students are encouraged to study a range of subjects early on, and then specialise later; however, this is as part of a four year undergraduate programme that starts at 17.  At Lancaster there is an element of this with students studying three subjects in their first year, but the three year degree programmes (normally starting at 18) means that for computing courses we now encourage students to take 2/3 of that first year in computing in order to lay sufficient ground to cover material in the rest of their course.  In most UK Universities there is less choice.

However, to be fair, the fault here is not simply that of university teaching and curricula; students seem less and less willing to take a wider view of their studies, indeed unwilling to consider anything that is not going to be marked for final assessment.  A five year old is not like this, and I assume this student resistance is the result of so many years in school, assessed and assessed since they are tiny; one of the reasons Fiona and I opted to home educate our own children (a right that seems often under threat, see “home education – let parents alone!“).  In fact, in the past there was greater degree of cross-curricula activity in British schools, but this was made far more difficult by the combination of the National Curriculum prescribing content,  SATs used for ‘ranking’ schools, and increasingly intrusive ‘quality’ and targets bureaucracy introduced from the 1980s onwards.

Paradoxically, once a student has chosen a particular discipline, we often then force a particular form of breadth within it.  Sometimes this is driven by external bodies, such as the BPA, which largely determines the curriculum in psychology courses across the UK.  However, we also do it within university departments as we determine what for us is considered a suitable spread of studies, and then forcing students into it no matter what their leanings and inclinations, and despite the fact that similar institutions may have completely different curricula.  So, when a student ‘fails’ a module they must retake the topic on which they are clearly struggling in order to scrape a pass or else ‘fail’ the entire course.  Instead surely we should use this this as an indication of aptitude and maybe instead allow students to take alternative modules in areas of strength.

Several colleagues at Talis are very interested in the Peer 2 Peer University (P2PU), which is attempting to create a much more student-led experience. I would guess that Stuart Hill might have greater sympathy with this endeavour, than with the traditional education system.  Personally, I have my doubts as to whether being virtually / digitally ‘in one spot‘ is the same as actually being co-present (but the OU manage), and whether being totally student-led looses the essence of scholarship, teaching1 and mentoring, which seems the essence of what a university should be. However, P2PU and similar forms of open education (such as the Khan Academy)  pose a serious intellectual challenge to the current academic system: Can we switch the balance back from assessment to education?  Can we enable students to find their true potential wherever it lies?

  1. Although ‘teaching’ is almost a dirty word now-a-days, perhaps I should write ‘facilitating learning’![back]

just counting

I have been an academic for 25 years and I am an academic by vocation, because it seems right, because God has given gifts and I try to see how to use them for the best. Of course vocation is only part of life and for around the first half or so of that time family took precedence over vocation, as is right.  However, for the past 8–10 years, academic life has been dominated by  teaching and in particular by the training, nurture and encouragement of research students.

It has been a privilege to see the growth and flowering of so many individuals.  At times hard, speaking soothing words whilst quaking within, Christmases with a cracker in one hand and a draft thesis in the other.  But it is amazing to see the transformations, like chrysalis into butterfly.

Recently, however, I was reading our faculty guidelines for workload allocation.  Activities are allocated to teaching, administration and research.  It transpires that the supervision of research students is not 50:50 research:training, not even 75:25, but wholly, only, and solely research.  That is all the nurture and training are null and void; research students are but fodder for my personal research agenda and the word ‘student’ a misnomer.

My head says this is simply counting, a simplification to make things add up, and yet my heart is saddened, sickened, broken by the notion as if ten years of my life were eaten and then spewed out.

It does not lessen the wonder I have seen in so many people and yet saddens me profoundly .

making part-time work?

Woke early worrying how to make the part-time thing work.

Looking forward through the year and adding up every odd day at home, still less than 18 weeks worth of ‘my time’, not exactly half of 52!  Even adding a couple of weeks of non-essential travel into ‘my’ budget doesn’t make it add up.

More worrying is that the time is all chopped up.  Just three solid months (and one of those is in July/August, maybe when I’d expect some research and holiday time anyway), the rest odd weeks split up with other commitments.  the model I’m aiming for is nearer the US 9 month contract idea with big periods for research, but struggling to keep blocks clear.

Also I’ve noticed myself allocating things that should be ‘university business’ to ‘my time’ as I know they won’t get done otherwise.  Got too used to doing the academic thing and planning time around assumption of 200% commitment averaging 80-90  hours a week.  Now trying to squash that into 50% of my time – no wonder it is difficult!

As the half pay cheques start to mount, I need to start to be ruthless.

Total Quality, Total Reward and Total Commitment

I’ve been reading bits of Richard Sennett’s The Craftsman1 off and on for some months. It has had many resonances, and I meant to write a post about it after reading its very first chapter. However, for now it is just part of one of the latter chapters that is fresh. Sennett refers to the work of W. Edwards Deming, the originator of the term ‘total quality control’. I was surprised at some of the quotes “The most important things cannot be measured”, “you can expect what you inspect” — in strong contrast to the metrics-based ‘quality’ that seems to pervade government thinking for many years whether it impacts health, policing or academia, and of course not unfamiliar to many in industry.

Continue reading

  1. Richard Sennett, The Craftsman, Penguin, 2009[back]

Been to London to visit the Queen

… well Queen Mary, University of London anyway. Giving a talk on “The New Media of Digital Light”1. While there I was given interesting tours of various research groups in the School of Electronic Engineering and Computer Science at QML including music that plays along with the drums, maps for the blind, and red dots on the heads of crowds at Covent Garden.

On the tube I noticed that if you are standing and look at the reflections in the curved tube train windows, all the seated passengers become Siamese twins joined at the head. Also looking down standing people tend to stand with their toes pointing outwards, whereas for seated people only the men do that. I feel there must be a social psychology paper in that, but it has probably already been written.

At the hotel neo-classical statues line the way down to an underground car park, and while seated at a WiFi sweet spot, was overhearing a dissident group planning a protest.

A typical day out in London.

A reminder of Wales, Aberavon Road, near QML

  1. work with Joe and Angie on Firefly technology[back]

now part-time!

Many people already knew this was happening, but for those that don’t — I am now officially a part-time university academic.

Now this does not mean I’m going to be a part-time academic, quite the opposite.  The reason for moving to working part-time at the University is to give me freedom to do the things I’d like to do as an academic, but never have time.  Including writing more, reading, and probably cutting some code!

Reading especially, and I don’t mean novels (although that would be nice), but journal papers and academic books.  Like most academics I know, for years I have only read things that I needed to review, assess, or comment on — or sometimes in a fretful rush, the day before a paper is due, scurried to find additional related literature that I should have known about anyway.  That is I’d like some time for scholarship!

I guess many people would find this odd: working full time for what sounds like doing your job anyway, but most academics will understand perfectly!

Practically, I will work at Lancaster in spurts of a few weeks, travel for meetings and things, sometimes from Lancs and sometimes direct from home, and when I am at home do a day a week on ‘normal’ academic things.

This does NOT mean I have more time to review, work on papers, or other academic things, but actually the opposite — this sort of thing needs to fit in my 50% paid time … so please don’t be offended or surprised if I say ‘no’ a little more.  The 50% of time that is not paid will be for special things I choose to do only — I have another employer — me 🙂

Watch my calendar to see what I am doing, but for periods marked @home, I may only pick up mail once a week on my ‘office day’.

Really doing this and keeping my normal academic things down to a manageable amount is going to be tough.  I have not managed to keep it to 100% of a sensible working week for years (usually more like 200%!).  However, I am hoping that the sight of the first few half pay cheques may strengthen my resolve 😉

In the immediate future, I am travelling or in Lancs for most of February and March with only about 2 weeks at home in between, however, April and first half of May I intend to be in Tiree watching the waves, and mainly writing about Physicality for the new Touch IT book.

On the edge: universities bureacratised to death?

Just took a quick peek at the new JISC report “Edgeless University: why higher education must embrace technology” prompted by a blog about it by Sarah Bartlett at Talis.

The report is set in the context of both an increasing number of overseas students, attracted by the UK’s educational reputation, and also the desire for widening access to universities.  I am not convinced by the idea that technology is necessarily the way to go for either of these goals as it is just so much harder and more expensive to produce good quality learning materials without massive economies of scale (as the OU has).  Also the report seems to mix up open access to research outputs and open access to learning.

However, it was not these issues, that caught my eye, but a quote by Thomas Kealey vice-chancellor of the University of Buckingham,  the UKs only private university.  For three years Buckingham has come top of UK student satisfaction surveys, and Kealey says:

This is the third year that we’ve come top because we are the only university in Britain that focuses on the student rather than on government or regulatory targets. (Edgeless University, p. 21)

Of course, those in the relevant departments of government would say that the regulations and targets are inteded to deliver education quality, but as so often this centralising of control, (started paradoxically in the UK during the Thatcher years), serves instead to constrain real quality that comes from people not rules.

In 1992 we saw the merging of the polytechnic and university sectors in the UK.  As well as diffferences in level of education, the former were tradtionally under the auspices of local goverment, whereas the latter were independent educational isntitutions. Those in the ex-polytechnic sector hoped to emulate the levels of attaiment and ethos of the older universities.  Instead, in recent years the whole sector seems to have been dragged down into a bureacratic mire where paper trails take precidence over students and scholarship.

Obviously private institutions, as  Kealey suggests, can escape this, but I hope that current and future government can have the foresight and humility to let go some of this centralised control, or risk destroying the very system it wishes to grow.

European working time directive 2012 – the end of the UK university?

Fiona @ lovefibre just forwarded me a link to a petition about retained firefighters, who evidently may be at risk as the right to opt out of European working time directive is rescinded.  Checking through to the Hansard record, it seems this is really a precautionary debate as the crunch is not until 2012.

However, I was wondering how that was going to impact UK academia if, in 2012, the 48 hour maximum cuts in.

It may make no difference if academics are not required to work more than 48 hours, just decide to do so voluntarily.  However, this presumably has all sorts of insurance ramifications – if we do a reference or paper outside the ‘official hours’ would we be covered by the University’s professional indemnity.  I guess also, in considering promotions and appointments, we would  have to ‘downgrade’ someone’s publications etc. to only include those that were done during paid working hours otherwise we would effectively be making the extra hours a requirement (as we currently do).

The university system has become totally dependent on these extra hours.  In a survey in the early 1990s the average hours worked were over 55 per week, and in the 15 years since then this has gone up substantially. I would guess now the average is well over 60, with many academics getting close to double the 48 hour maximum. I recall one colleague, who had recently had a baby, mentioning how he had cut back on work; now he stops work at 5pm … and doesn’t start again until 7:30pm, his ‘cut back’ week was still way in excess of 60 hours even with a young baby1. Worryingly this has spread beyond the academics and  departmental administrators are often at their desks at 7 or 8 o’clock in the evening, taking piles of work home and answering email through the weekend.  While I admire and appreciate their devotion, one has to wonder at the impact on their personal lives.

So, at a human level, enforcing limited working hours would be no bad thing; certainly many companies force this, forbidding work out of office hours.  However, practically speaking,  if the working time directive does become compulsory in 2012, I  cannot imagine how the University system could continue to function.

And … if you are planning to do a 3 year course, start now; who knows what things will be like after 3 years!

  1. Yea, and I know I can’t talk, as an inveterate workaholic I ‘cut back’ from a high of averaging 95 hours a few years ago and now try to keep around 80 max.  I was however very fortunate in that I was doing a PhD and then personal fellowships when our children were small, so was able to spend time with them and only later got mired in the academic quicksands.[back]