You would think crash reporting would be made as seamless and helpful as possible, after all your product has just failed in some way and you wish (a) to mollify the user; and (b) to solicit their assistance in obtaining a full report.
You would think …
In the following I will reflect on what goes wrong in Adobe’s crash reporting and some lessons we can learn from it.
In a previous post, Some lessons in extended interaction, courtesy Adobe, I described the tortuous process of obtaining Adobe Creative Suite 4. However, this done, on my return to the UK after trips to Rome and Italy, there was a nice box waiting for me and I installed it (or at least installed some bits, my disk is too full for it all!).
The main application I use is Dreamweaver, but I have been away again until, this week, so have not used it until today, except for checking it was running after the install. One of the reasons for upgrading to CS4 was that I had still been using CS2 and this was known to be unstable on Intel-based Macs. Dreamweaver has always crashed a lot (older versions NEVER exited without crashing), but I hoped that the newest version would be better … well I am always optimistic.
Of course, with one file open (my home page), and just plain HTML editing, sure enough after a few minutes the little spinning Mac rainbow wheel and a short while later Dreamweaver restarts and a crash report window appears.
Rather than simply saying “OK send the report”, I decided to be helpful and explain the full circumstances of the failure. At the bottom of the form it asked for an optional email if you were willing to be contacted by Adobe engineers should they needed more information. Still in a helpful mood I selected this and entered my email address. Then pressed “Submit” and got the following:
It said my email address was invalid, but it looked right. Maybe a comma instead of a full stop somewhere. I couldn’t see one, so copied and pasted from the settings in my email, just in case, but still the error message.
At this point I decided this was an interesting enough message to record, but when I navigated to another application in order to do the capture I lost sight of the crash report dialogue. I scanned the open application icons in the dock, but couldn’t see any that looked like a special crash reporter. I tried selecting Dreamweaver, itself but the dialogue wasn’t there. Happily, Mac has Exposé and I was able to see the crash dialogue there and so retrieve it.
After a bit I had an idea and entered a different, shorter email address – success!
The first email address I had entered was my university one, which both is an “ac.uk” domain and also has a departmental prefix: comp.lancs.ac.uk. I’m guessing the software was objecting to the four part domain name.
This is a simple dialogue box that (a) appears and (b) has a small number of fields to fill in. Two simple things and both failed as it (a) disappeared and (b) got the validation of the one complex field wrong!
So, what went wrong?
At one level there are a few superficial usability errors.
The error in (a) is a common one where a necessary window in some way ‘disappears’. I frequently get confused when the Mac reconnects to a WiFi network as there are two dialogue boxes: (i) one relates to the network connection, and (ii) the other is requesting requests a ‘keychain’ password so that the first can operate. Of course (i) cannot proceed until you have completed (ii), but (i) is bigger than (ii) and if you accidentally click the mouse over (i) then it completely obscures (ii)!
The error in (b) is even easier, just poor validation. In fact email addresses are very well specified in RFC 53221, so there is little excuse for getting them so wrong especially on a relatively simple domain name.
However, this also suggests process problems.
Looking at (a), this seems to be a failure in realistic testing. Crash reporting is a critical part of quality control and so one would expect to be subject to more rigorous testing, rather than less. To be fair I am sure it was well tested, but probably on a machine running only the software being test-crashed. The diappearing dialogue box problem only occurs in a machine under real use.
LESSON 1 : In testing have both standardised ‘clean’ situations, but also test in realistic scenarios (other applications, interruptions, cat running across the keyboard, etc.)
In an ideal world one should be able to verify that the dialogue box is available, but I know that never happens!
Looking at (b), this also is a test failure, after all surely Adobe has lists of millions of email addresses of people who have registered their products, why not pass a sample of these through the verification code. Instead the implementers will have written a small set of test cases, consisting of those kinds of email addresses they have thought of .
LESSON 2: Testing should attempt to go beyond preconceptions; after all if you have thought of the situation you have probably coded for it.
LESSON 3: Make use of available electronic data for automated testing.
However, (b) also points to a failure in reuse. When I registered the product I used the same email address that failed in the crash report. Clearly the verification code for registration is not the same as in the crash report – why?
LESSON 4: Reused code is less likely to have errors as it is tested in more situations.
So Adobe has once agian given us some useful usability lessons :-/
- Although when verifying email addresses, I usually use a slightly more conservative pattern than the RFC as there are some variants allowed in the full spec. that are never used … or will I one day be proved wrong! In particular, in my most permissive email checking I still add ‘/’ to the ‘special’ characters not allowed in a domain name, otherwise URLs could be read as one long domain name! [back]