Today, we’re going to look at a pilot study that UserZoom recently did for us. Based on my experience with that study, a few weeks ago, I gave them a 5-star review.

But first, a bit about how this all got started.

Originally, we touched base with UserZoom via Twitter and Oksana (who I’ve come to believe is either a part-time pro tennis player or figure skater), got back to us. I asked her if we could try out their software and that we’d love to have her or somebody else from UserZoom on the podcast. She put us in touch with their founder & CEO, Alfonso de la Nuez. He and I chatted a few times and ultimately decided to do a pilot study so we could get a feel for the software.

UserZoom is a suite of enterprise-level user testing tools. They’re really meant for Fortune 1000 companies. Their pricing structure is unique (read: $$$) but then again, as you’ll see, so is their product.

It’s also important to know what they are not.

UserZoom isn’t a user-testing company. They don’t have professionals on standby that you can hire to do your user testing. They are a SaaS solution.

That being said, for purposes of this pilot study, they did all the heavy lifting. All I had to do was give them a web address. And that was easy enough. Regular readers will recognize this one.

http://lessaccounting.com

They took it from there. A day later, Oksana sent me a link to their survey.

What a UserZoom User Test Looks Like

If you want to get an idea of what a UX researcher’s day looks like, watch the following video. Bring coffee.

In the following video, I simply go through the test. You’ll notice I run into a few minor problems but for the most part, things go off without a hitch.

If you don’t want to watch the video, here’s what happens:

At first, the survey asked me to install software to track my clicks. Despite the obvious privacy concerns, it didn’t tell me that I would be able to uninstall it later, and it didn’t really make it all that clear that I didn’t just put a key logger on my computer for all time. I didn’t feel great about this this installation but I decided to trust UserZoom to be a good neighbor. As somebody who knows that it’s possible to track clicks without this install, it was a bit weird. [Editor’s Note: Alfonso, in the comments of this post says that installing the plugin is completely optional.]

The genius of this method didn’t occur to me until Alfonso explained it later. By doing things this way, it’s possible to test any website without needing to add code to it first. This enables UserZoom users to test their competitor’s sites in addition to their own. Booyah.

The test itself was structured to give me a flavor for the different types of questions that could be asked. Since they designed the test without my input, it’s not meant to be taken as the pressing questions we have for LessAccounting. However, the survey does a good job of illustrating what you can do with the tool.

The survey asked the following questions:

1. What’s my age range?
2. Am I male or female?
3. I was shown a webpage for 7 seconds. Afterwards I was asked to say what I thought the website did.
4. I was asked to compare the homepage for both LessAccounting and Quickbooks and then choose the one whose demo I’d like to try.
5. They asked me to click on the screen about where I could find information for affiliate marketing.
6. They asked me to find out if LessAccounting works with Wells Fargo bank. I had a problem with this. The problem is that they place a second window at the bottom of the screen that contains both a “Success” and “Abandon” button. This is the window that I’m supposed to click to indicate success. But here were my problems:
   1. I felt confused when the click didn’t result in moving to the next page.
   2. It didn’t occur to me that I should be looking for a button in a different browser window. I actually closed the window and reentered the question. I mean, I was confused enough that I said “I think the right answer is to close the web browser.” Not good. But also not fatal. As we’ll see in the results, I was the only person who did this. Most of the other participants did not have this problem.
7. I had to find out how much LessAccounting charges for bookkeeping services. This was a navigation test. Once I found the answer, I clicked the “Success” button.
8. Choose how much LessAccounting charges for bookkeeping services from a multiple choice list. This is done to prove that I actually found the answer in the previous question.
9. On a scale of 1-7, how easy was it to find that information?
10. On a scale of 1-7 answer four more questions, all on the same page.
11. On a scale of 1-10 how likely it is I’d recommend the site to a friend?
12. What did I like the most about the website?
13. What did I like the least about the website?

After the test was over, I was pleasantly surprised to see that they let me uninstall the Google Chrome plugin that tracked my clicks. However I still wish they had informed me at the beginning of the survey that I would be able to remove it at the end of the process.

All in all, that’s a pretty smooth test. It’s an easy process to go through. I hit a few bumps but overall there were no fatal problems.

Now, let’s dig into the results.

The Results

After several days, at the conclusion of the study, Alfonso and I had a long conversation where he took me through the results. He went to some pains to make sure I understood that the results presented here are not scientifically valid. Their process for finding volunteers involved asking people to take the test via Twitter. They ended up getting 43 respondents, including myself.

 1. Demographic info

At the beginning of the test, I was asked my age and sex (young and frisky!). They do this because the survey can be constructed so that it’s a branching survey. What this means is that the answers to the questions can change which questions the user sees subsequently. A simple example of this would be if men were given one survey and women were given a different survey. But this can also be used for complex market segmentation.

2. System Information

UserZoom automatically tracks certain system information including:

1. Operating System
2. Browsers
3. Browser Resolution
4. Screen Resolution

 3. The 7-second Test

I was shown an image of the front page for seven seconds and was then asked to write down what I thought LessAccounting did.  In the results, they do something really neat. They give you the list of answers and they also break that information down into a tag cloud. This gives you an at-a-glance an idea of the common words that people use in their answer.

4. Pick One

In the next question I was asked to look at the front pages of two websites and to pick the one whose demo I’d rather try (based on what I saw on their website) and why. The results were nearly evenly split. But the explanations on why the user made the choice they did are enlightening (well, most of the time).

 5. Where My Affiliates At?

The next question asked me to click on the screen where I thought I would go to find more information about LessAccounting’s affiliate program. The results are provided in several ways: a clickmap, darkmap, or heatmap as well as the raw data.

Even with non-scientific data we can clearly see here that affiliate information is hard to find.

This data can be further understood because the test – when you make the wrong decision – asks you to explain why you clicked where you did. Let’s look at an example:

The original task stated: “Your friend told you that LessAccounting pays you for referring new users. Please click once on the screen where you would go to find more information about this.”

Seven people clicked the “Features” link in the menu. When asked why they thought that was the right place to click, here were their responses:

I think we can take away two things from their answers:

1. They’re confused
2. They’re guessing

To the extent that affiliate marketing is important to LessAccounting, with a 91% failure rate, their website clearly does not succeed at making that a selling point. Somebody would have to be determined to sign up as an affiliate if they ever expected to find the information.

 Tasks

In the survey, there were two tasks:

1. Find out if LessAccounting supports Wells Fargo bank
2. Find out how much their virtual bookkeeping software costs

I’m very familiar with the website and this graph above is worrisome, in particular the results to Task 1. A full 33% of people failed being able to find out if LessAccounting works with Wells Fargo. Scroll back up to the heatmap of the frontpage and look in the cloud of bank icons. Wells Fargo is right there. And 1/3 of the people who answered this question got it wrong. Talk about hidden in plain sight.

I’m less worried about Task 2 because I too think that it’s difficult to find the answer to that question. It doesn’t surprise me that nearly 2/3rd of respondents don’t know the answer. Instead it validates my suspicions.

For the purposes of this article, I’m only going to go through the first task in detail. This is the task that was surprising to me. But the other task also contains these same kinds of results.

Task 1: Finding information about Wells Fargo Bank

Clicks & Path

One of the crazy cool things that UserZoom does is track the paths through your site for a given task. The following picture shows the paths that participants took when trying to complete the task. This includes both paths and percentages.

I wish I could do it justice because in the Manager, you can zoom on on this image.

Effectiveness & Efficiency

These are the basic stats on how successful respondents were at finding out if LessAccounting works with Wells Fargo.

Validation Questionnaire

This is the question that was asked to everybody to validate that they indeed found the right answer to the question “Does LessAccounting.com work with Wells Fargo?”.

Success Questionnaire

This questionnaire was given to the people who got the answer right.

Error Questionnaire

This questionnaire was given to the people who got the answer wrong.

Abandon Questionnaire

This questionnaire was given to people who abandoned the task.

What we see from the results is that there’s a lack of clarity on how to find the answer. This is annoying since the Wells Fargo logo is on the front page but it’s also food for thought. Can we make the whole concept more clear?

The Final Questionnaire

These final four questions were asked to all of the users.

1. On a scale of 1-7 answer four more questions, all on the same page.
2. On a scale of 1-10 how likely it is I’d recommend the site to a friend?
3. What did I like the most about the website?
4. What did I like the least about the website?

I’ll show you the data and then we’ll talk about some conclusions.

Conclusions for LessAccounting

I have to say this again or Alfonso will think I didn’t hear him. These test results are not scientific. They should be taken as a group of 43 opinions that represent users who follow UserZoom on Twitter.

But, that being said, we believe in testing for “better” not testing for “best”. Improvement is improvement and the data here definitely tells us a few things. Chief among them are two main ideas:

1. The website is pretty
2. The website makes information hard to find

What’s funny about those conclusions is that this is the exact conversation I’m currently having with Allan about the new website design. But we’ll save that for a future post. The important thing to know is that this data – though not scientific – is relevant to current discussions involving a redesign.

Conclusions about UserZoom

As you can see, UserZoom gets at the answers to the pressing questions about your website. Perhaps its most special attribute is also its most subtle: it’s an all-in-one solution. We were able to ask a range of questions and to get valuable, actionable feedback.

Their enterprise tools – which I’ve not done the best job here in pointing out – really allow you to segment all of this data in ways that are meaningful to you. But hopefully you’ve gotten a good flavor for what their suite of tools has to offer.

Now imagine getting this type of data not only for your own website but for your competitor’s websites as well. It’s powerful stuff.