Usability Testing is Everybody’s Business

I’m guessing that if you’re reading this, you’re involved somehow in the business of developing technology.

In my experience, people who develop technology also tend to use a lot of it, some of it because they need to and some just because they enjoy it. They’re often early adopters, too.

So, odds are you personally have used a lot of technology. And you’ve probably noticed that a fair amount of it doesn’t work very well. Products are often confusing, or they throw error messages at you when you don’t think you’ve done anything wrong. Sometimes it turns out that you just can’t get there (your goal, like paying a bill online) from here.

Agile Australia 2016 attendees at Steve Krug’s keynote on 21 June 2016.

I often say that people in my line of work—usability—will always have a job, because it’s surprisingly hard (nearly impossible, really) to build something of any complexity that doesn’t have usability problems. The main reason is that when you’re building something you’re just too close to it. You know too much about it: You know how it’s supposed to work, and how it actually does work. As a result, it’s very hard to put yourself in the mindset of someone who doesn’t know how to use it, and to notice the things that are going to confuse them.

The net result is that people using the “stuff” we build often get stuck. Sometimes they complain, sometimes they give up quietly and try another product, and sometimes they just tough it out and make the best of it, even though they’re not very happy—or likely to buy a product from us again.

This has always been a problem, but with the rise of mobile apps it’s even worse. The evidence shows that most people trying an app with a price tag of 99 cents or free have a “one strike and you’re out” policy. Abandonment of mobile apps is huge.

In my 25 years as a usability consultant, I’ve become convinced that usability testing—which just means watching people try to use what you’re building—is the most effective and efficient way to improve the quality of your work…and to ensure that it actually ends up being used, not just gathering digital dust.

Usability testing has been around for quite a while now, and anyone who’s done it knows how effective it can be. There have always been two big problems with it, though: the usual suspects, time and money.

If you hire a consultant to do it for you, it will cost a lot of money. But it turns out it’s really not that hard to do, and almost anyone can do a pretty decent usability test themselves with just a little bit of instruction.

Steve Krug presents his keynote address at Agile Australia 2016

But if—as I recommend—you do it yourself, you run into the other problem: no one has time for it. Everyone—from coders to designers to product managers—always has more on their plate than there are hours in the day, and projects are always behind schedule, or at least teetering on the brink. Usability testing has always been in the category of one more thing, and not in the Steve Jobs “exciting surprise” sense, but in the “we just haven’t got time for one more thing” sense.

For some years now I’ve spent a lot of my time trying—in one way or another—to convince people who build digital things that they really ought to be doing usability testing, and showing them that it doesn’t have to be expensive and time-consuming.

I even wrote a usability testing handbook (Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems) in which my whole focus was making testing fast enough and simple enough that everyone can do it.

For an example of how my DIY “method” is simpler than traditional testing, consider the question of how many participants you need to use in each round of testing; a debate that has raged for decades in the usability community. The consensus generally has been that you need somewhere between 5 and 8 users, although some people argue for much larger—and more expensive—numbers for the sake of “statistical significance.”

After many years, I’ve settled on three users in each round of testing for a number of reasons:

• Finding three participants is less work than finding more, which is important if you’re doing the recruiting yourself.
• It’s much more important to do more rounds of testing than to wring everything you can out of each round. Testing with just a few users makes it easier to do more rounds.
• The first three users are very likely to encounter many of the most significant problems related to the tasks you’re testing.
• The biggest reason: even testing with three users will uncover more problems than you have the time and personnel to fix.

As a result, what’s important is not finding all of the problems, but finding the most significant ones and making sure you fix them as quickly as possible, then testing again.

You may be thinking, “That sounds pretty reasonable, but I know if I tried it I’d have people saying that if we’re only testing three people at a time we can’t prove anything, because it’s not statistically valid.”

As I said in Rocket Surgery:

Here’s what you should say to them:

“You’re absolutely right. Testing with so few people can’t possibly produce statistically valid results. The samples are way too small to even bother with statistics. But the point of this kind of testing isn’t to prove anything; the point is to identify major problems and make the thing better by fixing them. It just works, because most of the kinds of problems that need to be fixed are so obvious that there’s no need for ‘proof.’”

Try to say it with a lot of conviction and a friendly smile.

This article was originally published in AgileTODAY Magazine. You can subscribe free to this printed quarterly magazine.