It is easy to see why the usability community gets a bit obsessed by Jakob Nielsen - not only has he been around in the field for a long time, he writes so well - spinning a piece of research into wide- ranging conclusions pithily summarised. Sometimes I find myself thinking "Yes, but....you can't safely generalise so far from these results".
His recent Alertbox column, Variability in User Performance (May 15th 2006) is a case in point. It is based on some work he did in which people were asked to use Agere Systems website to find out the location of the corporate headquarters (they had to name the city). The quickest user took 28 seconds, the slowest took 420 seconds (though some could not complete the task at all). Looking at more typical users, Nielsen generalises:
When doing website tasks, the slowest 25% of users take 2.4 times as
long as the fastest 25% of users. This difference is much higher than
for other types of computer use; only programming shows a greater
disparity.
It would be very interesting to know what held up the slower users - did the website send them down one blind alley after another? Is the answer buried in long texts, so that differences in reading speed are a significant factor? Is there some luck involved? (For example, given this task I would expect the information to appear under either "Contact Us" or "About Us" - some research I did when working on the redesign of a corporate site last year suggests that these are about equally likely to be the place to find a postal address). Do some users have knowledge from which they can infer the answer - for example, if a telephone number but no postal address is given prominently on the site, some people might be able to infer the city from the area code of the phone number. Clearly the task is not easy, if even the star performer (or the luckiest test subject) takes 28 seconds.
Nielsen goes on to compare this fastest/slowest ration to those seen for other kinds of computing tasks and argues that more difficult tasks would produce a higher ration of slowest to fastest performance:
The more difficult a problem, the more individual differences we see.
As we approach the limits of human capabilities, the benefits of
additional brainpower -- mental abilities, talent, or whatever you want
to call it -- increase.
When using a website, for example, a user who can hold six chunks
of knowledge in short-term memory has great superiority over someone
who can hold only four chunks. The user with the better memory is less
likely to repeatedly go down the wrong path and more likely to
correctly assess how a given page relates to previous pages. In
contrast, a higher-capacity short-term memory doesn't help much in
simple text editing tasks, assuming you have a decent word processor
that doesn't require you to remember six things to move a paragraph.
He then argues that
This high variability is bad because it results from a degraded user experience for some people. After all, fast performance measures show that it's possible
to complete a website task within that time. Anything slower is a
result of users being delayed or sidetracked by usability problems. In
the perfect user interface, people should have no doubt about what to
do at any time and run no risk of making a wrong move. Given this, all
users would perform about the same, with only minor differences caused
by factors such as how fast they can click the mouse.
...Because website interfaces are challenging and must serve audiences
beyond the elite, it's particularly important that we tighten up the
Web user experience and reduce variability in user performance.
Yes, but. I think the argument that "high variability means poor usability" is bad. Imagine that we change Nielsen's test into a simpler one. Instead of asking people to find the information on a website, we give them copy from a corporate brochure of, say, about 1,000 words and ask them to find the information in that. People read at very different rates. A reading rate for "normal reading" is 100-200 words but speed reading can get you up to 700 words per minute - some people have managed 1,000 to 2,000 words per minute. One would be (un)lucky to unwittingly recruit champion speed readers in the trial, but I would expect that I'd get a big variability in how quickly my test users could use the 1,000 word brochure to come up with the answer. Clearly it would be simplistic to the point of meaninglessness to conclude that "reading is difficult" (assuming that I can get a variability of 3 or more I should conclude, according to Nielsen that reading is harder than programming). In fact, my fastest performers have learned tricks of speed reading that my other test users have not learned. This does not necessarily put the faster readers in the elite - it just says that they have troubled to acquire some speed reading skills, perhaps because they do a lot of Reading or are habitually impatient. Conversely, a low reading speed might not reflect low literacy, let alone low intelligence, mental abilities or other talent.
Extending this idea a bit further, one could try various designs of the brochure to see the usability effect. I would expect any of these to have an effect on how quickly the location of corporate headquarters was learned:
- Simpler language
- State location directly rather than indirectly ("Located in the capital of Scotland") or by inference
- typography, text layout and design
- Whether a picture of corporate headquarters is provided, and whether the city name appears in the caption
- Use or non-use of the word "headquarters" and its position in its paragraph (some speed readers skim paragraph starts, for example).
What is interesting is that these improvements might not have the effect of reducing the variability - some might assist the speed readers more than the normal readers, thus making the variability more extreme.
Also tucked away in Nielsen's argument is the assumption that slow = dysfunctional or frustrating. That could be the case of course - it would to know how the users felt about their task and the company website after their trial. I don't know whether users in Neilsen's study were asked to try and find the answer as quickly as they could - if not, there will be a factor in there that some folks like to work fast (due to habit, or to show off their computer prowess) while some are not in a hurry. In some designs speed does matter a lot (e.g. Customer Services are using the site to log phone calls with customers) in others, satisfaction is probably more important than speed and the two are not always going to be the same.
I don't claim that all this invalidates Neilsen's research study - just that one should not always extrapolate that high variability = difficult task, poor usability. It could mean that some of your trial subjects are tackling the task in a super-efficient (or super-inefficient) way you have not controlled for. I think the bad thing is not a high variability as such but whether it mean that too many would-be users are unable to use the site within a time and level of frustration they are comfortable with. Nobody would like a low-variability site that is low variabiliyt because it is awful for everyone!
We are used to this situation in download speeds, where someone with a broadband connection can download a page 20 times faster (or more) than someone on a dial-up connection. The variability there is beyond the web sites's control, but a good design tries to optimise the absolute time taken to download a page. If broadband users can see the page within one-hundredth of a second we are happy for them, but it hardly matters whether they see it after one-hundredth or one-thirtieth of a second (for most pourposes, both will seem instantaneous). We try, however to deliver the page to the dial-up user fast enough, even if that means efforts that the broadband guys will never notice.
It is good fun to try to find the location of Agere Systems HQ for yourself [caveat, I have no idea whether their website has changed since Nielsen's users used it, so my/your experience cannot be compared directly with theirs.] I think the answer is Allentown PA, USA - but it is far from easy to find this. Happy hunting!
