Citations and Sub-Area Bias in the UK Research Assessment Process

Alan Dix

This paper presents a citation-based analysis of selected results of REF2014, the periodic UK research assessment process. Data for the Computer Science and Informatics sub-panel includes ACM topic sub-area information, allowing a level of analysis hitherto impossible. While every effort is made during the REF process to be fair, the results suggest systematic latent bias may have emerged between sub-areas. Furthermore this may have had a systematic effect benefiting some institutions relative to others, and potentially also introducing gender bias. Metric-based analysis could in future be used as part of the human-assessment process to uncover and help eradicate latent bias.

Slides

References

1. Clerides, S., Pashardes, P. and Polycarpou, A. (2011) ‘Peer review vs metric-based assessment: testing for bias in the RAE ratings of UK economics departments’, Economica, vol. 78(311), pp. 565–83. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=982219
2. Lipsett , A. (2007). Institute criticises proposed RAE replacement. The Guardian, Thursday 13 December 2007. http://www.theguardian.com/education/2007/dec/13/
   researchassessmentexercise.highereducation
3. Oppenheim, C. (1995) The correlation between citation counts and the 1992 Research Assessment Exercises ratings for British library and information science departments, Journal of Documentation, 51:18-27.
4. Oppenheim, C. (1998) The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology, Journal of Documentation, 53:477-87. http://dois.mimas.ac.uk/DoIS/data/Articles/
   julkokltny:1998:v:54:i:5:p:477-487.html
5. Oppenheim, C. and Summers, M. (2008). Citation counts and the Research Assessment Exercise, part VI: Unit of assessment 67 (music). Information Research, 13(2), June 2008. http://www.informationr.net/ir/13-2/paper342.html
6. REF contextual data, 2013 (spreadsheet)
7. Research Excellence Framework 2014. http://www.ref.ac.uk/
8. Research Excellence Framework 2014 (2015). Output profiles and diversity. http://www.ref.ac.uk/results/analysis/outputprofilesanddiversity/
9. Research Excellence Framework 2014 (2015b). Results and submissions. http://results.ref.ac.uk
10. Research Excellence Framework 2014: Overview report by Main Panel B and Sub-panels 7 to 15, January 2015. http://www.ref.ac.uk/panels/paneloverviewreports/ panelminutes/
11. Sloman, M. (2015). Sub-panel 11 Computer-Science and Informatics REF Analysis.
12. Smith, A., and Eysenck, M. (2002) "The correlation between RAE ratings and citation counts in psychology," June 2002 http://psyserver.pc.rhbnc.ac.uk/citations.pdf

Figure 1. Sub-area REF 2014 profiles (excerpts from [10,11]).

Figure 2. REF 4* vs citation ranks.

https://alandix.com/academic/papers/WebSci2015-biblimetrics/

Alan Dix 10/1/2025