Usability Pitfalls and a Vision for the Future

By: Sandra Thwaites
Cognitive Systems Engineering

This is the third and final installment in the usability series. The first article provided a brief introduction to usability and an exercise to make the concepts more understandable. The second article reviewed the exercise in article one and suggested that the Neilsen’s 10 heuristics of usability might help a team of diverse professionals focus on some meaningful product attributes. This article will touch on two pitfalls of usability and will then address a vision for the future of usability by looking at some independent recent solutions.

TWO PITFALLS OF USABILITY
Usability is both an art and a science. The science elements include research methodology and statistics (e.g., concept of samples). The art includes good intuition about which usability evaluation method is best for a given situation and selecting the best core tasks for testing.  Two of the many pitfalls of usability are (1) evaluator or tester bias and (2) sample size or the number of subjects required for usability testing. 

Pitfall 1: Evaluator Effect
Since there is an ‘art’ element to usability, it should not be surprising that researchers have documented an ‘evaluator effect’. Hertzum and Jacobsen in 2003  and Jacobsen in 1998 are two papers that discuss the evaluator effect. Both reported that across practitioners there is a great deal of variation in the usability evaluation methods selected for a given usability test. Similarly, they found a great deal of variation in what practitioners identify as usability issues during testing. This was the case regardless of the practitioners' experience level! 

Findings of a so-called evaluator effect are, however, perhaps nothing more than validation for the tenant of psychology that says "all people are different."  Is there any profession in which all practitioners practice the same way? Medicine? Auto Mechanics? What seems essential is a trusting relationship between the buyer of the usability work and the usability professional. Usability professionals must be able to communicate their work and, if necessary, justifications for the way in which they will do that work.

Pitfall 2: Test Subject Sample Size issue
Recently, there has been controversy over the general rule that one need test only 5 to 7 subjects when examining usability. An article by Turner, Lewis and Nielsen in 2006 details calculations for determining the appropriate usability test sample size. They suggest that 5 to 7 subjects are sufficient provided that one is interested in finding only 4 usability problems.

In 2003, Carol Barnham  published a more accessible paper in the STC Usability SIG Newsletter, entitled “What’s in a Number?” trying to resolve the issue of whether 5 to 7 subjects are enough for usability testing.  Barnham concludes that the initial proposal of using 5 to 7 subjects is part of the “discount” method of usability. It makes usability testing affordable and is acceptable provided it follows the iterative (i.e., test-redesign-retest) method of development. This means that usability testing is meant to be done on more than one occasion during development.

A VISION FOR THE FUTURE OF USABILITY
If you think that usability is only about making screen-based interfaces work better, think again.  Usability principles are applied, or should be applied, to every service and product development.  Neither of the examples below relates to interface screens, but all clearly show the impact of usability.   

Getting usability out into the world at large is one grand hope this author has for the future. A few recent examples seem to hold the promise that usability can help make the world a more joyous and usable place.

Piano Stairs
One prime example of a noteworthy solution is the recent “piano stairs” project in Sweden. The video went ‘viral’ and shows a subway exit in Sweden, noting that most people exit via the escalator rather than the stairs. In an effort to get more people to use the stairs, they were turned into a piano of sorts.  Each stair emits a specific and different piano key sound when stepped on. There are even sharps and flats (black keys) positioned to one side of the stairs.

The video shows curious subway passengers noticing and trying-out the “new” stairs when exiting the subway. The video informs us that 66% more people take the stairs, after they were transformed into ‘piano stairs’. And, it is easy to see that the people using the stairs appear to be having a lot of fun. At the end of the video, we learn that Volkswagon sponsored this “experiment,” stating that "We believe that the easiest way to change people's behavior for the better is by making it fun to do."  This is usability at work.

Other projects are listed on the piano stairs link above such as the Merry-go-round pump project in Africa. Clean water is pumped from the ground as a result of children playing on a merry-go-round. This is another example of how ‘chores’ can be made to be fun. Video games are made to be fun and there is certainly no reason why other products and services should not also have fun as one of the usability goals. As long as someone on the development team is creative and can find a way to tie fun to the corporate bottom line then we can expect to see more fun in future products and services.

The Red Paperclip
Another story which received quite a lot of press, in Canada at least, became known as “the red paperclip”. A creative young man in his twenties had managed to trade-up his large red paperclip for a house! As stated succinctly on the individual’s website: “My name is Kyle MacDonald and I traded one red paperclip for a house. I started with one red paperclip on July 12 2005 and 14 trades later, on July 12, 2006 I traded with the Town of Kipling Saskatchewan for a house located at 503 Main Street.”  After two years of living in his house in a small Saskatchewan town, Kyle is apparently now eager to receive trade offers for his house.

The red paperclip story is just more proof that out of the box thinking can result in unbelievable things. As we move forward into the post credit crunch world of the unknown, it will take out of the box thinking and brave fresh minds to explore new avenues. Mankind can certainly do with projects that are more fun, more peaceful, more joyous, more healthful.

Usability is useless alone. It relies on visionaries and designers to spawn ideas. Together we can work on usability goals like sustainability, simplicity, fun, peacefulness, and neighborliness. Usability might just be a mindset and part of a consciousness ready to breathe. It is this author’s hope that the usability of tomorrow can play a role in unfolding a new, sustainable world.

Comments

[Caroline Jarrett]

I'm disappointed that you focus on 'pitfalls of usability' in this article.

First of all: evaluator effect. Yes, it is possible to mess up usability testing if you do any of these silly things:

- interrupt the user who is filling in the form
- show the user how to fill in the form at the start of the test (e.g. by 'demonstrating' the form)
- 'giving away' the 'right' answer by your body language or what you say
- becoming defensive about your form and arguing with the user.

BUT: if you are willing to be objective, swallow your pride, and sit quietly watching a real user filling in your form (that's all a usability test is), then you'll be amazed at how useful, easy, and informative usability testing is. As our usability hero Robert Barnett used to tell us constantly, you'll wonder why you haven't done more usability testing more often.

Secondly: Test subject sample size. (Aside: The term 'test subject' is outdated and no longer used by usability professionals. The 'subject' of your test is the form you're testing, not the users who are kindly giving up their time in return for a modest payment (or even just a thank-you) to test it for you). But let's talk about the sample size. The basic premise of the Turner, Lewis and Neilsen article is that they're talking about measuring usability and they're looking at the concept of how many problems you might find out of all the possible problems that there might be. OK, that's one aspect of the issue.

Now let's turn that around and look at what most of us want to do which is to *make our form better*. I can absolutely guarantee to you that in your first usability test with your first user, you'll find *at least one* serious and unexpected problem that will make you want to rush to change your form *right now*. If you test with three users, you'll have identified as many problems as you can cope with fixing. Fix those, and test again.

So don't worry about sample sizes or mathematics. Just try doing some usability testing. You definitely won't regret it.

And if you want a couple of books on this, first of all my book on forms has the basics of usability testing in our (very short) chapter "Usability testing: the best bit" - "Forms that work: Designing web forms for usability" www.formsthatwork.com or any good online or offline bookstore.

For book just on usability testing that I can heartily recommend, try Steve Krug's book "Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems". This is about testing web sites but is easily adapted to testing forms. It's a short, confidence-building book that you'll enjoy reading.