We Paid for User Testing, Here’s How it Went

User testing is one of those things people always tell you to do, but you probably never do it.

Testing is easy to put on the back burner as other more urgent issues pop up when building a website or product. It’s kind of expensive and time consuming, plus a lot of people don’t see the value.

We tried it, and let me tell you, it’s worth it.

If you aren’t familiar with user testing, it involves giving someone a certain task to perform, then watching them do it. They provide feedback as they go, but you don’t help them at all.

You could user test a new website design, a new product, or an app. An example could be “Visit our website, add a product to your cart, and checkout.” You would then watch them in person, or through screen capture video, and listen to their feedback. You may find that people have trouble finding the cart page, or they can’t use your website navigation properly.

One thing I can guarantee is that you will be surprised. People don’t use your website or product the way you think they will. They click the wrong buttons, don’t read instructional text, and miss totally obvious things (obvious to you at least).

We decided to do some user testing for our new product, WooCommerce Smart Checkout using usertesting.com. Here’s how it went.

UserTesting

There are lots of services out there to help you with user testing, you could even do it on your own. We went with usertesting.com, and it worked great for us.

How it works

To setup a test on usertesting.com, you can choose what type of testers you want (region, gender, age, device, etc). You create a set of instructions for them to follow, such as go to this page, click this button, etc.

The testers will then record themselves completing your instructions, by recording their screen and their audio.

We had users go to our test site, add an item to their cart, then checkout. We also had them go back and add a new item to the cart, and change the shipping options. Here’s a sample from our test:

What we found out

The bulk of the testing went well and most testers thought our product was very easy to use, which I’m happy with.

There were a few issues that came up, and it only took a couple of videos to get some valuable action items.

Address Autocomplete

WooCommerce Smart Checkout has an address autocomplete that is really neat. You start typing your address, and it searches nearby locations, and auto-fills all of the address fields. Many users remarked on how much they love this feature.

It works great, but one tester was confused that their address appeared on the page twice because of the autocomplete field.

We decided to move the autocomplete field into the first address field to eliminate this issue. That way their address is only shown once, and it is much more intuitive.

This may seem like an obvious fix to you, but as the developer of a site/product you can be blind to the way other people see things.

Saving Form Fields

We instructed people to fill out the checkout form, then go back and add another item to their cart. When they did this, the checkout form was blanked out, and this frustrated all of our users.

This may not be an issue on desktop, but iOS has auto-fill turned off by default. To fix this, we will experiment with temporarily saving form fields in the browser.

This is just the way forms work, so it did not occur to me to try to improve that. After watching testers struggle with this, it is a definite pain point, and something we can address in our product.

There were also a couple of other items, such as making it easier to go back to the cart page, and removing a scroll animation that didn’t work well.

Tips

There are a couple of things I would do differently for my next tests.

Give Good Instructions

When you are writing out your instructions, be specific and keep each one short.

Testers have to read your instructions and then execute the task, so don’t give them a bunch of stuff to memorize. You may want to use vague instructions so you can see if users struggle navigating your site or something, but in my case specific instructions worked better.

Ask for Feedback

Testers don’t know where you’re looking for feedback, so tell them. In my case, some testers were critiquing the website navigation and design, but all I cared about was the checkout process.

Get Your Money’s Worth

The tests are supposed to last about 15 minutes, so give people enough to do to fill up that time. Some of our tests only went 8 minutes or so, I could have had them click around a bit more to get more data.

Written Questions

Testers fill out some written questions after testing, I found that they just repeated the same things they said in the video. I would try to phrase my questions differently so I got new information next time.

Do you need user testing?

Yes. Do it.

At least give it a try so you can see what it’s like, I think you’ll be surprised.

You may not need to use a paid service, you could ask some friends or co-workers to help. The problem with asking friends or co-workers is that they are not a diverse sample, and you’ll probably procrastinate and never do it (like me).

The benefit of a paid service is that it’s fast and easy, and you get a wide variety of people from all over the world. You can target them by location, gender, age, and more, which gives you better data.

I can’t think of a better way to spend a few hundred bucks, especially if you are building a product or website that will be widely used.

What’s your experience with user testing?