{"id":39,"date":"2024-08-02T15:08:38","date_gmt":"2024-08-02T15:08:38","guid":{"rendered":"https:\/\/alandix.com\/ai4sj\/?page_id=39"},"modified":"2026-03-12T08:20:23","modified_gmt":"2026-03-12T08:20:23","slug":"mutant-algorithm-exams-during-covid","status":"publish","type":"page","link":"https:\/\/alandix.com\/ai4sj\/case-studies\/mutant-algorithm-exams-during-covid\/","title":{"rendered":"Mutant algorithm \u2013 exams during Covid"},"content":{"rendered":"

In Spring 2020, due to the Covid 19 pandemic, UK schools were closed to all except the children of key workers.\u00a0 It was decided early on that this would mean the normal A\u2019 level exams, administered at school leaving age (usually 18), would have to be cancelled.\u00a0 However, these exam results are used both for future jobs and university entrance.\u00a0 Ofqual<\/a>, the body responsible for overseeing UK exams, was tasked with coming up with an alternative way to deliver grades, based on teachers\u2019 predicted grades for their pupils, which are collected routinely every year to help universities make initial (usually conditional offers).\u00a0 However, it was known that predicted grades might overstate actual performance and so might not be fair both within the year (if different schools or teachers were more or less generous in their grading), or compared with previous years (if predicted grades were over-generous on average).<\/p>\n

Ofqual created an algorithm that combined various factors including the predicted grade and statistics on the performance of each school in previous years.\u00a0 This was then used to create a corrected grade that could be higher or lower than the teachers\u2019 predictions.<\/p>\n

When these predicted exam results were published in the summer of 2020, it created uproar.<\/p>\n

The system placed constraints on how many pupils could achieve certain grades and based its outputs on a schools’ prior performance, downgrading around 40 per cent of predicted results (although still allocating overall results far higher than previous years).\u00a0\u00a0 The data across centres\/schools was not equal and different centres had applied different principles to its standardization.<\/p>\n

Individual pupils felt that their grades had been unfairly reduced, sometimes substantially, and furthermore analysis of the overall effect showed it was operating unfairly between different parts of society.\u00a0 Overall, the algorithm allowed a level of \u2018grade inflation\u2019 compared with previous years, so that there were more top grades overall.\u00a0 However, analysis by journalists found that the increase amongst those going to fee-paying schools was around twice as high as that for state-funded schools [DM20<\/a>]; that is the algorithm seemed to entrench inequality<\/em>.<\/p>\n

\"\"<\/a><\/p>\n

There followed media furore [LS20<\/a>] and protests by pupils outside parliament [Co20<\/a>] Government ministers blamed Ofqual and the Prime Minister described it as \u2018mutant algorithm\u2019 [Co20<\/a>]. \u00a0Following the uproar from students and advocacy groups who took Ofqual and it\u2019s system to court [Fx20<\/a>] \u2013 the system was withdrawn and Ofqual ended up utilising the teachers\u2019 predicted grades. Ofqual later took the highly unusual step of publishing a 318 pages report, which provided explanations of the logics, goals and objective of the system [Of20<\/a>].<\/p>\n

The sad thing is that Ofqual were trying to be fair<\/em>.\u00a0 As soon as they knew the exams were to be cancelled, they gathered data on potential bias in teachers\u2019 predicted grades (aka. centre-assessed grades) and produced a public report on their findings [LW20<\/a>].\u00a0 This included some data from the UK, but also from other countries.\u00a0 The remarkable thing reading this report is how little data<\/em> they had to go on.\u00a0 One might have thought that the body responsible for oversight of qualifications would be constantly monitoring data about social, racial and gender equality, but in fact this was limited to a few targeted research projects.\u00a0 Possibly this is because the regular collection of protected characteristics would potentially be seen as problematic.<\/p>\n

Another factor evident in the whole process is a mutual lack of trust<\/em>.\u00a0 Government and Ofqual did not trust the teachers, the pupils did not trust Ofqual and the whole rigour of the exam system is in part built on distrust of the pupils themselves.\u00a0 Exam grades are effectively a replacement for personal relationships.<\/p>\n

Looking back at the algorithm itself.\u00a0 One of the problems was the unreasonable expectations<\/em> of the algorithm \u2013 it was being asked to do an intrinsically impossible task.\u00a0 Without the actual data on exam results, the belief was that it was possible to create an algorithm that would in a sense fairly (at an individual level) recreate the missing exam grades, whilst also being fair (in a group sense) within and between cohorts.\u00a0 The algorithm was expected to create data out of thin air.\u00a0\u00a0 It later transpired that early in the process Ofqual had realised this and advised government to scrap the exams entirely rather than attempt to create an algorithmic corrections, but were overruled.<\/p>\n

However, perhaps the most enlightening aspect of the story is the way in which the failings of the algorithm cast a mirror on the existing systems<\/em>.<\/p>\n

The Guardian article in August 2020 [DM20<\/a>] highlighted that the growth in top grades in independent (that is fee-paying) schools was 4.7% compared with 2% for state schools.\u00a0 However, this was an uplift to the previous year where around 45% of pupils in independent schools saw growth in top grades compared to around 20% of those at state schools, that is both had an uplift of 10% of their previous levels. \u00a0It wasn\u2019t so much that the algorithm was acting unfairly but that the existing A level system is deeply socially divisive.<\/p>\n

Finally, one of the reasons why this caused so much discord is that A level results are critically important to pupils.\u00a0 For those continuing to university the grades determine which university you can attend or whether you are admitted at all.\u00a0 For those going into the workplace, these are the grades that will be on their job applications for years to come.\u00a0 These grades quite literally shape the rest of the students\u2019 lives.\u00a0 Even in a normal year, a cold at the wrong time of year or a difficult time at home can make a difference to a student\u2019s life that will last years.\u00a0 This can be a personal tragedy for anyone, but also it is evident both from the Ofqual report and the raw grade figures that these impacts to not fall uniformly in society.<\/p>\n

As a postscript to this story, after the dust settled Ofqual updated the literature review from 2020.\u00a0 This revised report, based on UK and overseas evidence, concluded that centre-assessed grades were likely to disadvantage both poorer pupils and those with special educational needs [LN21<\/a>].\u00a0 Researchers at UCL also did a post-hic survey which highlighted the disproportionate impact of Covid on socially disadvantaged young people [AM21<\/a>].\u00a0 This survey included assessing the impact of the last-minute change from the Ofqual algorithm to using teachers\u2019 grades.\u00a0 It found that dropping the algorithm had led to 15% higher uplift for pupils with graduate parents [AM21b<\/a>] \u2013 that is dropping the algorithm had yet further increased social disparity.<\/p>\n

In other words, despite it being a bad idea in the first place, despite the public protests, government approbation and media vilification, Ofqual probably did a good job after all.<\/p>\n

References<\/h3>\n

[AM21<\/a>] Anders, J., Macmillan, L., Sturgis, P. and Wyness, G. (2021). Inequalities in young peoples\u2019 educational experiences and wellbeing during the Covid-19 pandemic (CEPEO Working Paper No. 21-08). Centre for Education Policy and Equalising Opportunities, UCL. https:\/\/EconPapers.repec.org\/RePEc:ucl:cepeow:21-08<\/a><\/p>\n

[AM21b<\/a>] Anders, J., Macmillan, L., Sturgis, P. and Wyness, G. (2021). The \u2018graduate parent\u2019 advantage in teacher assessed grades. (blog) UCL, Centre for Education Policy and Equalising Opportunities. 8 June 2021. https:\/\/blogs.ucl.ac.uk\/cepeo\/2021\/06\/08\/thegraduate-parentadvantageinteacherassessedgrades\/<\/a><\/p>\n

[Co20<\/a>] Sean Coughlan (2020). A-levels and GCSEs: Boris Johnson blames ‘mutant algorithm’ for exam fiasco. BBC News. 26 August 2020. https:\/\/www.bbc.co.uk\/news\/education-53923279<\/a><\/p>\n

[DM20<\/a>]\u00a0 Pamela Duncan, Niamh McIntyre, Rhi Storer and Cath Levett (2020).\u00a0 Who won and who lost: when A-levels meet the algorithm.\u00a0 The Guardian, 13 Aug 2020.\u00a0 https:\/\/www.theguardian.com\/education\/2020\/aug\/13\/who-won-and-who-lost-when-a-levels-meet-the-algorithm<\/a><\/p>\n

[Fx20<\/a>] Foxglove Legal (2020).\u00a0 We put a stop to the A Level grading algorithm!\u00a0 Foxglove Blog, 17th August 2020.\u00a0 https:\/\/www.foxglove.org.uk\/2020\/08\/17\/we-put-a-stop-to-the-a-level-grading-algorithm\/<\/a><\/p>\n

[LW20<\/a>] Ming Wei Lee and Merlin Walter (2020).\u00a0 Equality impact assessment: literature review. Office of Qualifications and Examinations Regulation (Ofqual) April 2020. https:\/\/assets.publishing.service.gov.uk\/media\/5e971f1de90e071a145ec51f\/Equality_impact_assessment_literature_review_15_April_2020.pdf<\/a><\/p>\n

[LN21<\/a>] Ming Wei Lee and Paul Newton (2021). Systematic divergence between teacher and test-based assessment. Office of Qualifications and Examinations Regulation (Ofqual), 17 May 2021.\u00a0 Ref. Ofqual\/21\/6781. https:\/\/www.gov.uk\/government\/publications\/systematic-divergence-between-teacher-and-test-based-assessment<\/a><\/p>\n

[LS20<\/a>] Rose Luckin, Anthony Seldon and Priya Lakhani (2020) Letter on\u00a0Inbuilt biases and the problem of algorithms.\u00a0 Guardian Letters. 17 Aug 2020. https:\/\/www.theguardian.com\/education\/2020\/aug\/17\/inbuilt-biases-and-the-problem-of-algorithms<\/a><\/p>\n

[Of20<\/a>] Ofqual (2020).\u00a0 Awarding GCSE, AS & A levels in summer 2020: interim report. Office of Qualifications and Examinations Regulation (Ofqual)13 August 2020. Ref: Ofqual\/20\/6656\/1. https:\/\/www.gov.uk\/government\/publications\/awarding-gcse-as-a-levels-in-summer-2020-interim-report<\/a><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

In Spring 2020, due to the Covid 19 pandemic, UK schools were closed to all except the children of key workers.\u00a0 It was decided early on that this would mean the normal A\u2019 level exams, administered at school leaving age (usually 18), would have to be cancelled.\u00a0 However, these exam results are used both for […]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":37,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"class_list":["post-39","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/pages\/39","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/comments?post=39"}],"version-history":[{"count":8,"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/pages\/39\/revisions"}],"predecessor-version":[{"id":86,"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/pages\/39\/revisions\/86"}],"up":[{"embeddable":true,"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/pages\/37"}],"wp:attachment":[{"href":"https:\/\/alandix.com\/ai4sj\/wp-json\/wp\/v2\/media?parent=39"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}