{"id":31,"date":"2015-05-09T19:15:21","date_gmt":"2015-05-09T19:15:21","guid":{"rendered":"https:\/\/alandix.com\/ref2014\/?page_id=31"},"modified":"2015-05-09T21:25:21","modified_gmt":"2015-05-09T21:25:21","slug":"results","status":"publish","type":"page","link":"https:\/\/alandix.com\/ref2014\/results\/","title":{"rendered":"results"},"content":{"rendered":"

Full results of citation analysis are in\u00a0REF-citation-vs-score-analysis-v5.0.xlsx<\/a><\/p>\n

The spreadsheet includes a description of the process used and a key of columns in the main results\u00a0worksheets.<\/p>\n

There are seven separate variants of the\u00a0analyses:<\/p>\n

    \n
  1. Scopus all years<\/li>\n
  2. Scopus 2008\u20132011<\/li>\n
  3. normalised 2008–2011<\/li>\n
  4. Google scholar all years with no citations as missing value<\/li>\n
  5. as (4) for 2008–2011<\/li>\n
  6. as (4) but with no citations treated as present and zero<\/li>\n
  7. as (6) for 2008–2011.<\/li>\n<\/ol>\n

    The choice of 2008–2011 is because outputs in 2012 and 2013 have relatively few citations, and hence 2008–2011 represents more reliable data.<\/p>\n

    Highlights — sub-area comparisons<\/h2>\n

    This is the first of the analysis tables. \u00a0Columns G–J are the quartile profile based on citations, with the upper quartile (best) in column J. Columns P–S are the REF profile form Sloman’s analysis (4* best). \u00a0Column V shows the ratio of the REF 4* column compared with\u00a0what would be predicted from\u00a0citations.<\/p>\n

    \"comparison-table-topics-2\"<\/a><\/p>\n

    Some\u00a0topics (such as Real Time Systems) have\u00a0small numbers of outputs so that\u00a0citation analysis will be less reliable. \u00a0However, ignoring these it can be seen that some topics (e.g. Logic) have many more 4* outputs than citation analysis would predict, whereas others (e.g. World Wide Web) have far\u00a0fewer. \u00a0The differences are extreme, up to 10:1.<\/p>\n

    Visualised in another way, the following diagram (prepared by Andrew Howes, who replicated the analysis using R), rank orders the topics using % of REF 4* outputs for vertical axis and % of upper quartile outputs for horizontal scale.<\/p>\n

    \"from-Andrew\"<\/a><\/p>\n

    It can be seen that there is effectively no correspondence\u00a0between REF score and citations. \u00a0The topics to the top left are those that are ranked more highly by REF than by citations, those\u00a0on the lower right are\u00a0ranked low by REF. \u00a0The split between more formal areas and more applied areas is evident.<\/p>\n

    Highlights — institutional (UoA) comparisons<\/h2>\n

    The following graph shows research power calculated both using REF profiles and citation data. \u00a0Research power is the GPA (average score) times the FTE of staff submitted. \u00a0For the citation data GPA is calculated\u00a0using predicted REF start scores.<\/p>\n

    \"institutions-power-cites-vs-ref\"<\/a><\/p>\n

     <\/p>\n

    It initially appears that there is a strong relationship between citations and REF scores at an institutional level, suggesting at first that the potential sub-area bias has ‘averaged out’ at the level of UoAs, many of which will contain a mix of research areas.<\/p>\n

    However, in the HEFCE funding model, funding is not allocated based on GPA, but heavily weighted towards 4* outputs with 2* and 1* outputs receiving no\u00a0funding at all. \u00a0To emulate this a revised measure based on a 7:3:0:0 weighting, called GPA# in reports, has been used. \u00a0The following graph shows the ‘power’ roughly proportional\u00a0to amount of money received, for this measure computed using actual REF profile and citation predictions.<\/p>\n

    \"inst-power-weighted\"<\/a><\/p>\n

    While there is still a rough correlation, some of the larger institutions appear to sit well above the line, and the spread in the lower group is quite large.<\/p>\n

    As an alternative view, the following shows\u00a0just GPA# (not FTE weighted) using REF scores and citations.<\/p>\n

    \"inst-gpa-weighted\"<\/a><\/p>\n

    Many\u00a0institutions have a relatively small number of outputs (around 100 is typical), so each point is not reliable. \u00a0However, the general trends are.<\/p>\n

    Of 27 institutions getting 25% or less than might be expected from citation analysis, 22 are post-1992 universities; and of 18 institutions getting 25% or more above\u00a0the citation prediction, 17 are pre-1992 universities.<\/p>\n

    There may be other effects, but as newer universities often have a more applied focus, this discrepancy may well be due to the sub-area differences. \u00a0This impression is reinforced by the fact that the two large institutions which appear as outliers on the GPA# power graph both have strong formal\/theoretical strands.<\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    Full results of citation analysis are in\u00a0REF-citation-vs-score-analysis-v5.0.xlsx The spreadsheet includes a description of the process used and a key of columns in the main results\u00a0worksheets. There are seven separate variants of the\u00a0analyses: Scopus all years Scopus 2008\u20132011 normalised 2008–2011 Google … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":3,"comment_status":"open","ping_status":"open","template":"","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"class_list":["post-31","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/pages\/31","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/comments?post=31"}],"version-history":[{"count":7,"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/pages\/31\/revisions"}],"predecessor-version":[{"id":53,"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/pages\/31\/revisions\/53"}],"wp:attachment":[{"href":"https:\/\/alandix.com\/ref2014\/wp-json\/wp\/v2\/media?parent=31"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}