For various reasons I won’t be at CHI1 in Barcelona, but I’d like to highlight two events I would have been part of had I been there.
One is more practice focused, the CHI 2026 UXR POV Workshop: Developing an AI-Powered UX Research Point of View (POV). (Thurs, 16th April, 14:15 – 15:45 CEST & 16:30 – 18:00 CEST). This workshop builds on a strand of work driven by Renée Barsoum, Huseyin Dogan and Stephen Griff, that seeks to create tools in the form of playcards, to help understand the wide range of POV of stakeholders during user research. I’ve made a short video for the workshop and I’ll distribute that after the event (no spoilers!).
The second is more research focused, a panel Does Peer Review Need to Change? A Panel on Reporting Standards and Checklists in the Age of AI (Mon, 13th April, 14:15 – 15:45 CEST). I’ll write a little more about this here, as I won’t be there in person, but these are my personal views the other panelists won’t necessarily agree! If you are in Barcelona, go to the panel to see what they say.
Why reporting standards?
The reason for this panel is that CHI along with many conferences faces issues of workload and consistency of reviews. The problems have been exacerbated by AI with both AI authored papers and AI reviews.
This is not just a problem for CHI. Some years ago a computing conference2 needed to split its programme committee into two halves to deal with the volume of papers. They were worried about consistency between the sub-committees, so had both sub-committees look at an overlapping sample of the papers. They found that the two sub-committees agreed on a small number of very high quality papers and also a larger number of definite rejects. However between these extremes, the large majority of the papers, the agreement was no higher than chance.
The CHI panel will describe the way that some other disciplines have tried to tackle this. This has been particularly important in medicine, where rigour in research is literally a life-or-death issue; there there are standards for different kinds of work, for example, the CONSORT standards for reporting randomised trials. For other disciplines, including education and psychology, it can be hard to agree on definitions of quality, so they have often opted for standard ways to present results, making it easier for reviewers to focus on specific aspects, and hence lead to more consistent reviews. Could a similar approach work in CHI?
A launch pad not a shackle
One of the reasons I was invited onto the panel was because of a CHI paper a few years ago HARK No More: On the Preregistration of CHI Experiments with Andy Cockburn and Carl Gutwin. Although I was the card-carrying mathematician/statistician amongst the authors, I was also the one who kicked back slightly against strict demands for pre-registration. Instead I advocated using it as base point from which variations in data collection or analysis might be made, but where such variations needed to be clearly and strongly justified.
Similar caution is needed with standardised reporting more broadly. Even with a range of different templates for different kinds of papers, there will always be work that doesn’t quite fit … I’m wondering what reporting standards for pictorials would look like! So any process should allow variations and papers that completely step outside the accepted formats – otherwise the discipline will be frozen. But, when the standards are not followed, the discrepancies need to be justified and the bar set higher.
Democratising access
While the reasons for considering reporting standards emerge from issues such as workload and consistency of reviewing, the greatest benefits in my mind are far wider. One of these is to help open up venues to those who are not part of in-groups. During 40 years of publishing I have seen my own papers grow in length, with massively more references per paper, but not convinced that more recent work is more informative.
A year or two ago ACM surveyed members on acceptable uses of AI in academic publishing: should it be allowed at all, should it be allowable to include an AI in the author list? After a point my answers became variants of a single theme: “if we can’t tell the difference between AI bullshit and academic bullshit, AI is not the problem”.
CHI especially has a genre, a way of writing, which successful CHI authors learn and share through apprenticeship among their colleagues and students. It is not that the substance doesn’t matter, but there are particular ways to say it as well. More formulaic paper structures would help authors focus on the content, rather than the form, making it easier for readers new to the community to draw out the critical information, and helping ensure that high quality work of authors new to the community is recognised.
Building the discipline
Academic venues are often rated based on their acceptance rates, with around 25% being the mark of a good venue. One of my comments in discussing the panel proposal (with which none of the other panelists agree!) was3:
a successful discipline has a 100% acceptance rate
Of course I don’t mean just accept everything, but rather that a 25% accept rate means 75% of work is effectively wasted. Now of course some of that will get published elsewhere, and not all work will be equally informative or innovative, but if academics and researchers are spending time on work that is effectively thrown away, that is a disaster. Ideally every piece of research work should be of a form and standard that contributes to knowledge even if incrementally. If this is not the case, then the discipline has a duty to educate researchers, especially early career researchers.
Reporting standards could help. As well as retrospectively asking, “how do I write the work I have done better?”, they can be used prospectively to plan, “what work do I need to do in order to be able to write a paper of this form”. That is templates for good reporting become templates for good research raising the overall quality of the discipline.
That seems a goal worth pursuing.
- CHI is the largest international conferences in human–computer interaction.[back]
- I can’t recall which conference this was, if you know please let me know.[back]
- I’m not entirely alone however, it has been suggested that low acceptance rates might reduce the overall quality of the conference! B. Parhami, “Low Acceptance Rates of Conference Papers Considered Harmful” in Computer, vol. 49, no. 04, pp. 70-73, Apr. 2016, doi:10.1109/MC.2016.106.[back]