A/B Testing Basics

There are a lot of reasons that people leave websites but the largest one has to be that the website is too hard to use. A truism of web design is that the design should seek to keep the cognitive-load of the user to a minimum.

Steve Krug even named his first book after this fact. “Don’t Make Me Think!”

What this means is that users have a low tolerance for figuring out how a website works. They want to get to the information they are there to find. They don’t want to actually find it. They really would rather the information be so easy to get to that it’s like it just appeared.

A/B Testing is most effective when it’s used to reduce friction on the critical path.

What is A/B Testing?

A/B testing is the process of testing two or more variations of a webpage or an element of a webpage to see which one achieves your page goal most frequently. It’s possible to test whole pages or distinct elements within the page.

KISSmetrics has a great graphic to illustrate possible testing areas on a page.

What Should I Measure?

The key to A/B testing is to test an item that impacts the flow of the website’s critical path or the website’s conversion rate. We divide the critical path into 4-segments: filters, objects, confirmers, and goals. Each segment along the path is defined by what the user is doing at that point in the path.

Filter: In this stage of the path, a user is trying to find the thing that brought them to your site. That thing is the object. At this stage of the game, it’s best to make it as quick and easy for a user to get to the object as possible. [highlight color=”eg. yellow, black”]Focus your A/B Tests on making your choices clear and reduce your bounce rate as much as possible.[/highlight]

Object: This is the reason the user is on your site. The goal of this page is to get somebody to add the item to the cart. [highlight color=”eg. yellow, black”]Test items that influence this goal. This includes, headlines, body copy, product images, and button placement.[/highlight]

Confirmers: All of the pages that come after the object page but before the goal page are confirming pages. Their job is to subtly indicate to the user that they are making the right decision. In an e-commerce site this would be the cart and the checkout process. This entire process has to (a) not get in the way or be so complex as to make users leave the site and (b) it has to offer up details that the user will wonder about their order: shipping policy, return policy, security, etc.

These pages are also part of your conversion funnel. Look at exit rates on these pages in particular to figure out which pages to test so that the flow is through the funnel and that you’re not losing people unnecessarily in the process. [highlight color=”eg. yellow, black”]Test elements on the page that increase movement through the conversion funnel.[/highlight]

This is probably best done in conjunction with an Analytics program such as Google Analytics that can track goal funnels and a screen capture tool like Inspectlet. These additional tools can pinpoint what you should be testing in your A/B tests and the addition of watching your shoppers will provide clues as to what your “B” in the A/B test should be.

Goals: This is the one page you don’t have to test. It’s the goal page. When users get here, [highlight color=”eg. yellow, black”]you win[/highlight].

Keys to A/B Testing

Know why you are running your A/B test.
The item being tested should be noticeable to your audience and should impact your page goals.
Test only one variable at a time. If you test more than one variable (multivariate testing) you are going to have problems knowing for sure what variable caused the change.
Your test needs to be statistically significant.

When we say ‘statistically significant’, what we mean is that you need to have a large enough sample size so that the test can be proved to be valid within a certain margin of error. We call this our ‘confidence rate’. In order to believe the results of the test, you want to [highlight color=”eg. yellow, black”]look for a 95% confidence rate[/highlight].

The easiest way to determine how large you need your sample size to be in order to reach a 95% confidence rate is to use one of the many online calculators to do the heavy lifting for you. The Split Test calculator from usereffect.com is an easy way to see how many people you need to run your test. There are also tons of other sample size calculators out there.

Conducting an A/B Test

To conduct an A/B test on your website, you’re going to need to use an online tool. Whichtestwon.com has a list of 40 A/B testing tools.

There are essentially two ways these tools will conduct an A/B test.

They will test two separate pages
They will use javascript to conduct the A/B test within the page

The first way requires you to have two completely different URLs for the pages that you are testing. Something along the lines of http://domain.com/test1.html and http://domain.com/test2.html.

The second way requires that you place some javascript on your website so that it can dynamically serve one option or the other.

Which method you use depends on which tool you use. Both of them will give you valid results but implementing one or the other requires a different amount of setup time.