I gave a panel presentation “Metrics are rubbish, but …” at the Association of Research Managers and Administrators Annual Conference, Liverpool, 7th June 2017.

The panel, “Doing metrics responsibly: how far have we come and what are the next steps“,  was organised by the ARMA Metrics Special Interest Group, and brought together speakers from industry, government and academia, as part of a discussion on the appropriate use of metrics,

.. and, not surprisingly give this micro-site, although my talk started “metrics are rubbish, but …”, it finished, “… people are far worse”!

Although this mini-site mainly talks about the dangers of human processes, there are of course many dangers of using metrics as a form of research assessment.

An article in the Journal of Electronic Publishing, “Counting research ⇒ directing research. The hazard of using simple metrics to evaluate scientific contributions. An EU experience.” by Kai Olsen and Alessio Malizia looks at this with a pan-European perspective, in particular comping the very inconsistent ways different countries use metrics.

As well as different ways of using information, the authors show how different countries may regard the same venue very differently.  This is particularly significant in countries which use venue as the main criteria directly against the recommendations for good practice in The San Francisco Declaration on Research Assessment (DORA).

LSE impact blog: “Evaluating research assessment: Metrics-based analysis exposes implicit bias in REF2014 results” based on the REF metrics analysis reported in this micro-site.

A series of blog posts go step by step through the main outcomes of the post-REF citation analysis:

• REF Redux 1 – UK research assessment for computing; what it means and is it right?
• REF Redux 2 – world ranking of UK computing
• REF Redux 3 – plain citations
• REF Redux 4 – institutional effects
• REF Redux 5 – growing the gender gap
• REF Redux 6 — reasons and remedies

On 16th September I gave a talk ‘Data Matters’ at Bournemouth University.  The main topic was the REF data analysis.

Abstract

For half of this talk I’ll present work I’ve been doing this year on post-hoc analysis of REF data for computing. As a member of SP11 I feel some responsibility for the results (!). I have been comparing REF results with large-scale bibliometrics to see whether the process, which attempted to be as fair and transparent as possible, managed to attain that goal. The results are worrying, suggesting that some areas of computing and some types of institution may well have been disadvantaged relative to others. This has implications for future funding, for the prospects of women in computing and for the UK economy. Some of this relates to specific socio-technical issues in SP11 processes, but some have lessons for other subject areas.

For the rest of the talk I will give a brief tour of projects I’m involved with where HCI and computing is being used to address data needs of other people, in particular what I have previously called the ‘long tail of small data’.

In the InConcert project we’ve been looking at various Musicological data sets seeking to re-imagine archival processes in ways which preserve underlying academic values, and yet leverage computational power to allow researchers to use their professional judgement in new ways. In Open Data Islands and Communities, the question is to what extent open data can be of value to small communities (rural or urban), both as consumers and providers. In my work at Talis, we’ve been considering the value of detailed learning analytics to help individual academcis and students, but also understanding how this can fit within the clamour of daily academic life. Finally, form my 2013 walk around Wales I have loads of ‘quantified self’ data including the largest public domain ECG trace, and facing challenges of data documentation and dissemination.

Common themes are around the way numerous small heterogeneous datasets can be brought together, the need to understand data in context, the politics of data, and the extent to which we do and don’t value data in academic scholarship.

I will be a panel member on HCI and the REF, alongside David Benyon, Tom Rodden and Peter Bath at the British HCI conference, in Lincoln, 15th July 2015.   Come along if you want to hear about the implications of REF 2014 for HCI research in the UK and even if you are not from the UK.  Many countries are looking towards REF as an example of how to do University research assessment; so the lessons from UK REF may well be the way things are done in your country soon!

I’ll be presenting a paper “Citations and Sub-Area Bias in the UK Research Assessment Process” on aspects of this analysis at the workshop on “Quantifying and Analysing Scholarly Communication on the Web (ASCW’15)” at WebSci 2015 on June 30th in Oxford.

There is an online discussion at the workshop site, so please discuss 🙂