We Paid for User Testing, Here’s How it Went

User testing is one of those things people always tell you to do, but you probably never do it.

Testing is easy to put on the back burner as other more urgent issues pop up when building a website or product. It’s kind of expensive and time consuming, plus a lot of people don’t see the value.

We tried it, and let me tell you, it’s worth it.

If you aren’t familiar with user testing, it involves giving someone a certain task to perform, then watching them do it. They provide feedback as they go, but you don’t help them at all.

You could user test a new website design, a new product, or an app. An example could be “Visit our website, add a product to your cart, and checkout.” You would then watch them in person, or through screen capture video, and listen to their feedback. You may find that people have trouble finding the cart page, or they can’t use your website navigation properly.

One thing I can guarantee is that you will be surprised. People don’t use your website or product the way you think they will. They click the wrong buttons, don’t read instructional text, and miss totally obvious things (obvious to you at least).

We decided to do some user testing for our new product, WooCommerce Smart Checkout using usertesting.com. Here’s how it went.

UserTesting

User Testing

There are lots of services out there to help you with user testing, you could even do it on your own. We went with usertesting.com, and it worked great for us.

How it works

Screen Shot 2016-04-25 at 3.12.16 PM

To setup a test on usertesting.com, you can choose what type of testers you want (region, gender, age, device, etc). You create a set of instructions for them to follow, such as go to this page, click this button, etc.

The testers will then record themselves completing your instructions, by recording their screen and their audio.

We had users go to our test site, add an item to their cart, then checkout. We also had them go back and add a new item to the cart, and change the shipping options. Here’s a sample from our test:

What we found out

The bulk of the testing went well and most testers thought our product was very easy to use, which I’m happy with.

There were a few issues that came up, and it only took a couple of videos to get some valuable action items.

Address Autocomplete

WooCommerce Smart Checkout has an address autocomplete that is really neat. You start typing your address, and it searches nearby locations, and auto-fills all of the address fields. Many users remarked on how much they love this feature.

address-autofill

It works great, but one tester was confused that their address appeared on the page twice because of the autocomplete field.

Screen Shot 2016-04-25 at 3.31.10 PM

We decided to move the autocomplete field into the first address field to eliminate this issue. That way their address is only shown once, and it is much more intuitive.

address-auto2up

This may seem like an obvious fix to you, but as the developer of a site/product you can be blind to the way other people see things.

Saving Form Fields

We instructed people to fill out the checkout form, then go back and add another item to their cart. When they did this, the checkout form was blanked out, and this frustrated all of our users.

This may not be an issue on desktop, but iOS has auto-fill turned off by default. To fix this, we will experiment with temporarily saving form fields in the browser.

This is just the way forms work, so it did not occur to me to try to improve that. After watching testers struggle with this, it is a definite pain point, and something we can address in our product.

There were also a couple of other items, such as making it easier to go back to the cart page, and removing a scroll animation that didn’t work well.

Tips

There are a couple of things I would do differently for my next tests.

Give Good Instructions

When you are writing out your instructions, be specific and keep each one short.

Testers have to read your instructions and then execute the task, so don’t give them a bunch of stuff to memorize. You may want to use vague instructions so you can see if users struggle navigating your site or something, but in my case specific instructions worked better.

Ask for Feedback

Testers don’t know where you’re looking for feedback, so tell them. In my case, some testers were critiquing the website navigation and design, but all I cared about was the checkout process.

Get Your Money’s Worth

The tests are supposed to last about 15 minutes, so give people enough to do to fill up that time. Some of our tests only went 8 minutes or so, I could have had them click around a bit more to get more data.

Written Questions

Testers fill out some written questions after testing, I found that they just repeated the same things they said in the video. I would try to phrase my questions differently so I got new information next time.

Do you need user testing?

Yes. Do it.

At least give it a try so you can see what it’s like, I think you’ll be surprised.

You may not need to use a paid service, you could ask some friends or co-workers to help. The problem with asking friends or co-workers is that they are not a diverse sample, and you’ll probably procrastinate and never do it (like me).

The benefit of a paid service is that it’s fast and easy, and you get a wide variety of people from all over the world. You can target them by location, gender, age, and more, which gives you better data.

I can’t think of a better way to spend a few hundred bucks, especially if you are building a product or website that will be widely used.

What’s your experience with user testing?

8 Comments

  1. Matt Pritchett on April 27, 2016 at 8:28 am

    While I certainly agree that user testing is awesome, $49 per video for 10 videos and then $99/video is crazy expensive for a fledgling business, which is where this is most needed, in my opinion. 🙁

    • Scott Bolinger on April 27, 2016 at 10:50 am

      Ya it’s not cheap, but even at that price I still think it’s worth it for any business. You could also user test on your own for free, it’s just more work.

    • Preston on February 15, 2017 at 7:17 am

      Why not try an alternative? UserTest.io offers desktop user testing for $13-$23 per video! You can check them out at: https://usertest.io

  2. Wesley Wiseman on May 2, 2016 at 6:02 pm

    Well, I agree with Matt. That’s a crap load of money and I’m not sure it’s worth it. Can’t hurt I suppose, except for the bank account.

  3. Alejandro Rivas on February 12, 2017 at 9:01 am

    Usertesting is definitely needed. It is however expensive but the data coming back being collected can be worth the investment. Thank you.

  4. johnny chen on February 12, 2017 at 9:03 am

    We paid for user testing before and it is costly. But the results that came back was surprising for us and we were able to make small changes that made big differences.

  5. Hub Wired Media on June 13, 2017 at 7:26 pm

    Hi Would you please help me by sharing who are the best developers you used? I need some for my project.

  6. Armando Orta on July 18, 2018 at 12:53 am

    Great blog post! I love the idea of user testing please keep up the good work.

Leave a Comment





Scott Bolinger