Choosing an A/B Testing Tool for Small Business

The choice to use a A/B testing can be hard. Terminology is the first thing to get past – see previous post. Once you figure that out (the secret is to clarify what you WANT, not what they HAVE), my advice is to layout everything in a chart to see how the tools stack up. That’s what this post is about.

I took the perspective of a small business owner with less than $50 dollars a month for A/B testing.

We need to know the basics – name, URL, cost to start and if they have a free trial period. Next, we need to know about the reporting features. And, if they test multiple pages or just a single page of the site.

And, finally, we need judge if we trust the developers and the tool. It all comes down to trust. Do you trust they will deliver what you need and limit frustration. Do they speak my language? Do they understand my situation and goals? Are they legit?

I took a few hours to survey the tools.

A/B testing services: (y-axis, if you will)

Here are the attributes: (x-axis, if you will)

Name
Address
Cost
Free trial?
Same page?
Reporting?
Trust?
Description

You can view the actual “UX Tool Matrix” here on Google Docs.

let’s go surveying…

Google Website Optimizer

Google webapp for A/B testing. The variations are created by developing a separate page and giving that URL and the original URL to Google. Lastly, you need need a ‘conversion’ URL. Whole bunch of URLs, seems to be powerful and simple, but perhaps not as straight forward as Optimizely. Free to use.

Optimizely

Webapp that states “A/B testing you’ll actually use”. It copies the page from a URL and then uses a script to substitute out the different variations. You can create the variations in a slick, in-broswer editor. I get the feeling this is like Greasemonkey scripts that alter the page for the visitor. Neat idea. My concern is for display issues and incompatibilities. Starts at 17/ month after a 30 day free trial.

Visual Website Optimizer

Visual Website Optimizer is similar to Optimizely – almost a clone (not sure which is the clone). Seems well done. They doing a good job of speaking directly to my needs and I trust them. It’s the most expensive of the group – starting at $49/month.

HeadMetrics

Another webapp that is similar to Visual Website Optimizer and Optimizely. You can start right away by entering the URL, make variations via the editor, and -after you sign up- conduct a test.

The problem here is they don’t seem very “present”. The lack of social media is concerning and I don’t really trust them. Starts at 6.97 /month and 30 day trial.

Split Testing Pro

A desktop application that does A/B testing. At first blush, I was all in. This tool could be great. It has a low price, non-subscription based and gives me… wait. There is a real, info-merical, squeeze page vibe going on here that is off putting. If you are willing to risk it, $199 (but wait, today only $149 – gimme a break)

Webtrends

Webtrends is an outlier in this group. It is a comprehensive service – not just A/B Testing. It doesn’t SAY A/B testing on the homepage – it is mentioned in the ‘Optimization’ section. The prices are not listed. There is no product tour. I got nothing.

Summary: Which tools can I suggest?

I can’t. I don’t have a recommendation. But, I can say that Ben and I will investigate Optimizely and Google Website optimizer next week. The plan is use those two tools on a site and relate our experience. I hopeful for much more insight.