Alan Dix > HCI Education

hard questions have no answers

SIGCHI Bulletin HCI Education column September 2000. bulletin (pdf)

Although it will be a new academic year when this issue comes out, at the time of writing it is in the middle of examination board season in the UK. A few weeks ago, final-year students sweated in examination halls up and down the country, but they have now finished and are drinking wine, having all-night parties, falling in rivers and in love, savouring the sweet taste of freedom. Whilst they relax in the sun, in darkened rooms academics sit huddled over endless piles of papers and spreadsheets deciding their fate: who fails, who passes, the honours and the wooden spoon. Whereas the Greek Fates, Clotho, Lachesis and Atropos, spun and measured out men's lives with thread and then cast it on the fire, here the future lives of young women and men are reduced to lists of numbers and grades.

In education we literally play with peoples lives.

I hate setting exams and hate even more marking them. Partly because it's a lot of work, but more because I dread that final judgement. Formative testing is hard work, but you are giving feedback to grow your students. Exams just label them.

One thing that makes me feel better about marking exams is my knowledge of statistics. I know that my marking is not totally consistent, the marks at the start of the pile are not the same as those at the end of the pile. If I do half the marking one day and half the next it will probably matter whether I got a good night's sleep in between. Although we make every effort to avoid it, marking is inevitably a process with a large random element. However, the nice thing about random effects is that they average out to give a true reading. Each exam has many questions, each question has several parts, each course may have several assessments and the eventual degree is based on many exams and other assessments.

One accidental benefit of changing institutions mid-year has been (shh! don't tell my colleagues) no exams to mark! (To be fair I did set some at Staffordshire before I left and left extensive sample answers, so I haven't been entirely negligent.)

Although I've escaped exam boards at my own institution, I haven't escaped entirely. In the UK we have a system of 'external examiners'. The old saying is "who polices the policeman". Well in UK academic circles the answer is "the external examiner" I don't know if the same holds true in other countries. The external examiner's job is partly to verify that judgements are arrived at in a fair and unbiased fashion and partly to ensure a (probably mythical) consistency of standards between institutions.

Consistency of standards - is a degree in institution A the same 'standard' as one in institution B? In countries with a single nationally prescribed syllabus across all institutions this is perhaps easier to assess, but for those where every course is different? It's a bit like fairness within an exam. It is possible to be totally fair and totally consistent, simply have a universal syllabus, and test it with multiple choice questions. Very fair, consistent and totally useless at measuring anything of value. And of course if we really are worried about a fair and consistent educational system, it may be worth getting rid of all teachers (must make a difference) and even doing something about poverty, health provision and early educational access

Probably the most interesting of my external appointments is as "Chief External Examiner of the Post-Graduate Modular Scheme" at Cheltenham and Gloucester College. Each subject has its own individual exam board overseen by external examiners in its own discipline and I get to attend cross-disciplinary boards which receive the results of the subject boards and award Masters degrees based largely on formulae and recommendations from the relevant representative. Sounds exciting huh? Happily everyone there agrees and the box ticking parts happen as quickly as possible only stopping to consider the difficult cases.

It is of course always these exceptional, marginal cases which are most challenging and lead to broader questions. What does it mean to have a 'Masters' degree when you consider subjects as diverse as Fine Art, Computing and Theology? Consistency is one thing within a subject, but can we have a touchstone for 'post-graduateness' that covers this sort of range? Phrases like 'analytic approach', 'maturity', 'reflection' spring to mind, but it is not clear whether any such list is universally applicable and even less whether they can be interpreted uniformly! And, after that, capturing this in rules - rules that ensure levels of consistency whilst still allowing appropriate discretion. Sounds familiar it's an archetypal socio-technical design problem assigning roles to the human and automatic parts of a system so that the whole works together. Such systems won't be free of human frailty, bias and mistakes, but will benefit, when rules don't apply, from our unique abilities to make hard judgements and, of course, our willingness to accept responsibility for them.