Figure 3. A Web-based shared calendar system, Lotus SmartSuite's Organizer 97©.
This paper explores the implications of critical performance parameters for the design of Web-based applications. It suggests that transfer of software to the Web may introduce problems in application domains that are performance-sensitive. It offers examples of such application domains, including airline reservation, medical record-keeping and calendar maintenance, and explores possible critical parameters to take into consideration. It concludes with some suggestions on viewing system design in terms of performance, and on the need to consider performance in conjunction with other system requirements.
The Web's susceptibility to response delays and access failures is a matter for widespread concern. For this very reason, however, it may lead ultimately to benefits, for it may alert the IT community to some serious weaknesses in current design practice. In particular, it may heighten concern as to whether computers deliver the productivity improvements they are claimed to offer, a concern that has been voiced by Landauer (1995). Problems with the Web may also force us to ask whether we have the tools and methods to address temporal issues in the design of interactive systems (Johnson and Gray 1996).
A particular issue, which I attempt to address in this paper, is how we can design future generations of interactive systems to deliver better performance than their predecessors. This has not yet become a serious issue in the context of the Web. However, if networked web-style computing gains acceptance as a paradigm in which to develop application software, it will begin to supersede existing platforms such as DOS and Windows. Questions of whether performance gains are being achieved and whether productivity is thus being enhanced will start to acquire vital importance. My concern is that this issue has arisen in the past, for example with the superseding of DOS by Windows, and has not been properly addressed. We do not know whether Windows-based applications perform any better, for their users, than their DOS-based predecessors. For the same reasons, we will find it hard to tell whether the Web provides better performance than either Windows or DOS.
I believe that we have been prevented from making these kinds of performance comparisons between interactive systems because we have not identified the application-specific critical parameters that govern their design. If we could identify such parameters we could use them to assess how well a proposed system design would serve its purpose, or to compare two competing designs. Until now, research in Human Computer Interaction has been conducted largely without reference to critical parameters, primarily because these parameters are difficult to identify (Newman 97). In this paper I will argue that critical parameters for interactive applications can be identified, and that the prospect of widespread Web-based computing makes it imperative that we should move towards identifying them and using them more in our design work.
Critical parameters
Critical parameters, or figures of merit, are established
parameters by which designers predict or measure how well a system or
artefact serves its designated purpose. A critical parameter is a
performance parameter of a particular kind. It applies to
technologies that serve a particular purpose, e.g., supporting
Customer Service operations or assisting the gathering of criminal
evidence. It is an established performance parameter, applied
routinely and without question to the design of technologies for the
stated purpose. It is an abstraction or approximation of all of the
performance parameters that might be applied, enabling the designer
to compare alternative designs, and perhaps choose between them, on
the basis of quick estimates rather than an exhaustive evaluations.
And, finally, it is a parameter that is manipulable by the designer:
a design can be produced, with some degree of certainty, that will
meet a pre-defined performance parameter.
Many examples of the use of critical parameters can be found in the world of engineering. For example, traffic intersections are designed primarily on the basis of two parameters, capacity and delay. Airports are designed in terms of the number of operations (take-offs and landings) that can be handled per hour or per day. Oil wells are designed to deliver a certain number of barrels per day of oil. In each of these fields of design there is a conspicuous common culture: a culture of universally accepted requirements parameters; of comparing between, and selecting from, alternative forms of solution; of making these comparisons on the basis of performance; of re-using past solutions rather than starting from scratch; and of aiming for (and usually hitting) specific performance targets; in short, a culture of engineering design.
In contrast, the design of interactive systems, as described in the literature, shows an almost complete disregard for critical parameters. Instead the critical design property is the system's functionality (Adelson and Soloway 1985). Design focuses on a single selected form of solution, often a deliberate departure from forms used in the past (Newman 1994). Requirements are drawn up primarily in terms of the functionality that the system should deliver. The chosen solution is adapted to the requirements by a series of enhancements, fixes, performance improvements and, occasionally, simplificationspduring which the requirements themselves are susceptible to drift. Simulations and evaluations of the design focus almost entirely on finding flaws in the design; and when quantitative measurements are carried out, they usually involve metrics chosen at the time of the experiment rather than at the outset of design (Newman 1997).
Nevertheless we can find a few documented instances of design problems for which critical parameters have been identified. Heath et al. (1993) have described operations in equity dealing, where there is a legal requirement to record all deals withing one minute; they discuss the problem of designing systems to meet this performance target. The well-known work of Gray et al. (1993) on modelling the performance of tasks by telephone operatorspthe so-called "Project Ernestine"prelied on the identification of call-handling time as the main critical parameter in design. More recently Tweedie et al. (1996) have presented different methods of interactive visualisation, comparing them in terms a critical parameter, the time taken by users to perform a light-bulb design task.
How might the existence of known critical parameters change the way we design interactive systems? I see evidence in these few examples, and in my own attempts at design on this basis, of a natural tendency to design more in accordance with an engineering culture. In particular, there is an incentive to compare alternative solution strategies, rather than adopt a single strategy and make it work. There is also a greater advantage to be gained from re-using existing strategies. Meanwhile the design task provides a context for carrying out analytical research, in the manner described by Rogers in his discussion of engineering practice (Rogers 1983). For example, research may be needed to refine the definition of a critical parameter or, as in Project Ernestine, to develop a more powerful model for predicting performance. The prospect of integrating research more tightly into design brings with it the possibility of a new, more central role for HCI in system design.
Some examples
To highlight the implications of designing in terms of critical
parameters, I will offer three examples. They are chosen to
illustrate the general opportunity that critical parameters provide,
and the specific problems presented by new platforms, such as the
Web. The three application domains are airline reservation,
appointment calendars and general medical practice.
Airline Reservation
Airline reservation is a particularly long-established application of interactive technology. The first online reservation system, American Airlines' Sabre, went into service in 1963 and is still one of two or three leading systems used by the travel industry (Copeland 1995). Airline reservation is particularly relevant to discussions of Web performance, because the industry is moving towards increasing direct bookings by travellers; the Web provides a natural basis for such a booking service.
Airline reservation systems are typically designed as a "star" network with a computational hub providing service to many thousands of users through a massive global communication network. The standard user interface presented to the agent is a relatively primitive command-line interface, as shown in Figure 1. Data are presented in compact formats, using codes that agents must learn by heart. Command syntax is also compact and highly arcane, as might be expected of a system that has evolved over three decades. Training costs for these systems are high, and this creates problems in an industry where profit margins are under threat and staff turnover is on the increase.
103JANLGWMAD9A 03JAN THU LGW/Z MAD/#1 1BA 464 C4 M4 S4 B4 L4 Q4 V4 LGWMAD 0830 1150 73S 0 2DA 683 C4 S4 M0 Q0 LGWMAD 1000 1320 B11 L 0 X26 3IB 609 C4 M4 L4 K4 LGWMAD 1130 1435 737 0 4BA 466 C4 M4 S4 B4 L4 Q4 V4 LGWMAD 1615 1935 73S 0 5DA 689 C4 S4 M0 Q0 LGWMAD 1855 2210 B11 D 0 X6 VA*4 03JAN DPTR ARVL MEALS S EQP ELPD ACCUM MILES SMK LGW MAD 1615P 1935P D 73S 3.20 3.20 1740 N
Figure 1. American Airlines' Sabre system: typical commands (shown here in bold) and display formats. The operator has asked to view flights (code key 1) between Gatwick and Madrid on January 3rd, leaving after 9 a.m., and has then asked to verify flight information (code key VA) for the fourth flight listed.
Travel agencies are attempting to increase productivity by a variety of means that include simplifying the user interface, e.g., by the provision of scripts or canned dialogues that prompt the agent for answers to questions: date, departure city, arrival city, etc. While this reduces training costs it appears to save agents little time; scripts are not popular with agents who have learned the "native" user interface. Meanwhile the airlines are encouraging travellers to bypass the travel agencies and make their own bookings and ticket purchases online, and thus avoid payment of commissions by the airlines.
It seems possible, and perhaps even likely, that all airline reservation services will gravitate towards the Web, for airline staff, agents and public alike. Public services are already moving in this direction: Figure 2 shows EasySabre, a public version of Sabre. EasySabre's command syntax and display formats are simpler than native Sabre's but still show a strong family resemblance, as can be seen. Meanwhile travel agencies are beginning to move onto Windows platforms. However, the attractions of moving directly to the Web must be apparent: the Web offers a standard protocol that can be received by a wide range of terminals, and a style of interaction that is becoming familiar to more and more computer users.
FLIGHT AVAILABILITY
From: (LGW) GATWICK-LONDON, UK To: (MAD) MADRID, SP THURSDAY JAN-03 --------------------------------------------------------------------- Flight Leave Arrive Meal ST FC Equip Classes of Service 1 BA 464 LGW 830A MAD 1150A B 0 @ 73S C M S B L Q V 2 DA 683 LGW 1000A MAD 120P L 0 @ B11 M Q 3 IB 609 LGW 1130A MAD 235P L 0 737 C M L K 4 BA 466 LGW 415P MAD 735P D 0 @ 73S C M S B L Q V 5 DA 689 LGW 655P MAD 1010P D 1 @ B11 C S M Q
Figure 2. A display generated by EasySabre, an airline reservation service available over the Web. The same information has been requested as in Figure 1, this time by typing in /air,lgw,mad,03jan,9a.
The advantages of Web-based airline reservation could of course be offset by reduced performance. But would users notice? Travel agents are accustomed to degradations in response from the network hub, and might not notice the effect of network delays. In any case, system response time is not necessarily a critical parameter. The travel agent's time is basically at the disposal of the traveller making the inquiry, and if a transaction takes longer than usually it is the traveller who suffers. It appears hard to identify a critical parameter for the design of reservation systems for travel agents' use; or, for that matter, for systems used by the public.
On the other hand the use of reservation systems by airline check-in staff does sometimes take place in situations where transaction time is critical. One such situation arises when travellers have been delayed and, having missed their connections, need to be booked quickly onto the next available flight. In this context the time to rebook is a critical design parameter. If Web-based technologies are used in systems for airline staff, they will need to be tuned for performance in order to avoid slowing down the rebooking task. Overall, airline reservation is an application area where new software platforms may bring performance degradations, which in turn affect the service provided to the traveller.
Appointments Calendars
The management of an appointments calendar is a much more familiar application than either the previous example or the next one. Indeed virtually everybody in professional life must keep a calendar (or diary) to avoid missing appointments. Any technology that offers to assist calendar management has a large potential user population. Thus far the main contenders have been workstation-based shared calendar systems and personal calendars running on pocket computers.
The Web has an opportunity to become a dominant platform for online calendars, for it offers the combination of portability and connectivity that many calendar users seek. At present, users basically face a choice between pocket calendars (both paper and electronic) that can be taken anywhere but are then disconnected from the rest of the world; and online group calendars that can be accessed only at a workstation. Of course, not everybody demands portability: two thirds of the participants in Payne's published study left their calendars in their offices at all times (Payne 1993). But a Web-based shared calendar, suitable for use on a networked pocket computer, might appear capable of satisfying all users. Systems of this kind are starting to be made available (see Figure 3).
Figure 3. A Web-based
shared calendar system, Lotus SmartSuite's Organizer
97©.
One problem with this scenario lies, I believe, in the need to achieve certain performance demands of calendar use. A particularly critical parameter is the time available to two people, in face-to-face or telephone contact, to schedule an appointment. Stated another way, how much time can reasonably elapse between Person A saying to Person B, "Can we meet next Wednesday afternoon?" and Person B responding? At what point does Person B, still waiting for the calendar system to respond, give up and suggest calling Person A back later?
In the days before electronic calendars existed, the answer to such a request from Person A was usually given as soon as the requisite page had been found. The only reason for delay in responding would have been a lost calendar. The arrival of electronic calendars has brought with it the phenomenon of slow calendar accesspa delay of ten or more seconds while the calendar program is invoked. Now things appear to be deteriorating still further. My personal experience recently has suggested that calendar access may sometimes be so slow or unreliable that the appointment cannot be made on the spot; Person B must arrange to contact Person A later when the calendar in accessible. I do not know the value of the critical parameter in question, but would guess at a maximum delay of 20 seconds, after which Person B will feel obliged to give up. This level of performance may be hard to guarantee in Web-based calendar systems.
Supporting the General Practitioner
As a rule neither airline reservation nor appointments scheduling are life-or-death activities. My third application domainpgeneral practicep is different, however. The use of computers in medical consultation, introduced into the British health service over the last decade, gives cause for concern on a number of grounds. Greatbatch et al. (1993) have described how computers disrupt doctor-patient dialogue in ways that may inhibit the patient from disclosing their health concerns. Meanwhile questions are being raised about the quality of the notes taken by GPs on their surgery computers, and the validity of using these notes as data for health surveys. Overall, we may wonder whether GP systems help or hinder the doctor's work, since notetaking appears to take just as long as before computerization, or even longer. When doctors are under pressure to improve their quality of service within the 7 or 8 minutes average consultation time, they must be concerned about these issues.
GP support might seem a relatively unlikely application for the Web. However, two factors suggest that a transition to the Web may be likely:
Figure 4. A typical GP
system's display of a patient's notes. From Allan D. and Quinlan C.
(1995) Making Sense of Computers in General Practice. Oxford:
Radcliffe Medical Press.
It is too early to say how current GP systems might make the transition to the Web. At the very least the system must continue to support the two main functions of current systems, i.e., online notes and prescription preparation. These functions, now performed locally on the GP's computer, may not run as fast over the Web. There may be several side-effects as a result: the length of each consultation may stretch from the current 7 or 8 minutes up to 10 minutes or more; the GP's attention may be focused even more on the computer, and less on the patient, than it is at present; or the GP will have to rush through the consultation in order to maintain current work rates. We need to understand the critical parameters of general practice better, so that we can design the next generation of GP systems, Web-based or not, to offer better levels of support than at present.
My primary purpose in introducing and illustrating the issue of critical performance parameters is to encourage discussion. Many different attitudes and approaches can be taken to this basic issue in the context of the Web. One set of views, which I will offer here as my own particular standpoint, runs as follows:
I accept that this is only one of a number of viewpoints on system design. Another viewpoint, widespread in the interactive systems business, is that achieving functionality and quality matters more than achieving performance. In other words, the designer's first imperative is to overcome functional shortcomings of previous systems. This leads directly to a second imperative: to remove as many functional faults as possible from the new system, from both the user interface and the supporting software. This is the focus of nearly all evaluation techniques currently practised by user interface designers, and it is also the focus of methods for improving software quality. Of course, removing faults from software may result in improving performance, but this cannot be guaranteed.
A third point of view places emphasis on market opportunities. It holds that the emergence of the Web has made possible radically new ways of information sharing, of collaboration, of advertising, of entertainment. These have introduced a paradigm shift in computer use, making it difficult if not impossible to compare performance of tasks under pre- and post-Web conditions. Instead the important measure, from this viewpoint, is business growth. How will established industries such as travel and medicine be affected by increased access to global information? What new businesses will be made possible, and what existing businesses will go away?
HCI researchers need to find a role that is relevant in the context of any or all of these points of view. They need to know how to achieve performance targets, how to provide new functionality without new design faults, and how to help design systems that will achieve commercial success. My concern is that the tools available for achieving performance targets are inadequate, and so the designer of Web-based applications may produce systems that perform worse than DOS- or Windows-based applications. We need to make sure that the growth of the Web does not bring with it a decline in users' productivity. If more attention is paid to application-specific critical parameters this may be avoided.
Adelson B. and Soloway E. (1985) "The Role of Domain Experience in Software Design." IEEE Trans. on Software Engineering, Vol SE-11, no. 11 (November 1985), pp. 1351-1360.
Copeland D. G., Mason R. O. and McKenney J. L. (1995) "Sabre: The Development of Information-Based Competence and Execution of Information-Based Competition." IEEE Annals of the History of Computing, Vol. 17 no 3, pp 30-57.
Gray W. D., John B. E. and Atwood M. E. (1993). "Project Ernestine: Validating a Goms Analysis for Predicting and Explaining Real-World Task Performance." Human Computer Interaction Vol 8, pp 237-309.
Greatbatch D., Luff P., Heath C. and Campion P. (1993) "Interpersonal Communication and Human-Computer Interaction: An Examination of the Use of Computers in Medical Consultations." Intnl. J. of Interacting with Computers, Vol. 5, pp. 193-216.
Heath C., Jirotka M., Luff P. and Hindmarsh J. (1993). "Unpacking Collaboration: The Interactional Organisation of Trading in a City Dealing Room." Proc. Third European Conf. on Computer-Supported Cooperative Work &endash; ECSCW '93. Kluwer, pp. 155-170.
Johnson C. and Gray P., eds. (1996) "Temporal Aspects of Usability: Papers from a Workshop." ACM SIGCHI Bulletin, vol. 28 (2), pp. 32-56.
Landauer T. K. (1995) The Trouble with Computers. Cambridge MA: MIT Press.
Newman W. M. (1994). "A Preliminary Analysis of the Products of HCI Research, Based on Pro Forma Abstracts." Proceedings of CHI '94 Human Factors in Computing Systems (April 24-28, Boston, MA) ACM/SIGCHI, N.Y., pp. 278-284.
Newman W. M. (1997) "Better or Just Different? On the Benefits of Designing Interactive Systems in terms of Critical Parameters." Proc. DIS '97, Designing Interactive Systems (Amsterdam, 18-20 August), in press.
Payne S. J. (1993) "Understanding Calendar Use." Human-Computer Interaction, Vol 8., pp 83-100.
Rogers G. F. C. (1983) The Nature of Engineering: a Philosophy of Technology. London: Macmillan.
Tweedie L., Spence R., Dawkes H. and Hua S. (1996) "Externalising Abstract Mathematical Models" Proceedings of CHI '96 Human Factors in Computing Systems (April 13-18, Vancouver BC) ACM/SIGCHI, N.Y., pp. 406-412.